Panoramic surround view image generation method, device and vehicle

Abstract

The present application discloses a method, device and vehicle for generating panoramic surround view images, and belongs to the field of vehicle technology. The method for generating panoramic surround view images includes: determining obstacle projection information and a fusion area between target adjacent environment images in multiple environment images based on multiple environment images of a target vehicle at multiple perspectives; generating an initial stitching path based on the fusion area; determining a target stitching path based on the initial stitching path and obstacle projection information; processing target adjacent environment images based on the target stitching path to obtain a panoramic surround view image. The method for generating panoramic surround view images of the present application can more flexibly avoid multiple obstacle projections to obtain a stitching path, and has a better effect of eliminating blind spots, thereby helping to improve the user experience and ensure driving safety; and has a wide range of applicable scenarios.

Description

Technical Field

[0001] The present application belongs to the field of vehicle technology, and in particular, relates to a method, device and vehicle for generating a panoramic surround view image. Background Art

[0002] Around View Monitor (AVM) is part of the automatic parking system. It is a highly practical function that can greatly improve user experience and intelligent driving safety. In related technologies, a panoramic image around the vehicle body is mainly obtained by stitching four-way surround fisheye lens images; however, this method has limited effect on reducing blind spots, thus affecting user experience and driving safety. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a method, device and vehicle for generating panoramic surround images, which can more flexibly avoid multiple obstacle projections to obtain a spliced ​​path, have a better blind spot elimination effect, thereby helping to improve the user experience and ensure driving safety; and has a wide range of applicable scenarios.

[0004] In a first aspect, the present application provides a method for generating a panoramic surround image, the method comprising:

[0005] Determining, based on a plurality of environment images of the target vehicle at a plurality of viewing angles, a fusion region between obstacle projection information and adjacent environment images of the target in the plurality of environment images;

[0006] Based on the fusion region, generating an initial splicing path;

[0007] Determining a target stitching path based on the initial stitching path and the obstacle projection information;

[0008] The target adjacent environment image is processed based on the target stitching path to obtain a panoramic surround image.

[0009] According to the panoramic surround image generation method of the present application, by combining the initial stitching path generated based on the fusion area with the obstacle projection information to determine the target stitching path corresponding to the target adjacent environment image, it is possible to more flexibly avoid multiple obstacle projections to obtain a stitching path, and has a better blind spot elimination effect, thereby helping to improve the user experience and ensure driving safety; and it has a wide range of applicable scenarios.

[0010] According to an embodiment of the present application, determining a target stitching path based on the initial stitching path and the obstacle projection information includes:

[0011] In a case where it is determined based on the obstacle projection information that at least one obstacle projection intersects the initial splicing path, the target splicing path is determined based on the driving parameters of the target vehicle.

[0012] According to an embodiment of the present application, determining the target splicing path based on the driving parameters of the target vehicle includes:

[0013] Based on the driving parameters, determining a predicted driving path corresponding to the target vehicle;

[0014] The target splicing path is determined based on the intersection of the predicted driving path and the fusion area.

[0015] According to an embodiment of the present application, determining the target splicing path based on the intersection of the predicted driving path and the fusion area includes:

[0016] In a case where the predicted driving path and the fusion area do not intersect, the target splicing path is determined based on at least one of a priority corresponding to each obstacle projection and a distance between each obstacle projection and the target vehicle.

[0017] According to an embodiment of the present application, determining the target splicing path based on the intersection of the predicted driving path and the fusion area includes:

[0018] In a case where the predicted driving path and the fusion area intersect, the target splicing path is determined based on the position information of the obstacle projection and the position information of the predicted driving path.

[0019] According to an embodiment of the present application, determining the target splicing path based on the position information of the obstacle projection and the position information of the predicted driving path includes:

[0020] Based on the position information of the obstacle projection and the position information of the predicted driving path, determining the intersection of the obstacle projection and the predicted driving path;

[0021] The target splicing path is determined according to the intersection of the obstacle projection and the predicted driving path.

[0022] According to an embodiment of the present application, determining the target splicing path based on the position information of the obstacle projection and the position information of the predicted driving path includes:

[0023] When at least one obstacle projection among all the obstacle projections does not intersect with the predicted driving path, deleting the obstacle projection that does not intersect with the predicted driving path;

[0024] The target splicing path is determined based on the deleted obstacle projection information.

[0025] According to an embodiment of the present application, determining the target stitching path based on the deleted obstacle projection information includes:

[0026] When it is determined based on the deleted obstacle projection information that there is no remaining obstacle projection, the target splicing path is determined based on the fusion area.

[0027] According to an embodiment of the present application, determining the target stitching path based on the deleted obstacle projection information includes:

[0028] In a case where it is determined based on the deleted obstacle projection information that there are remaining obstacle projections, the target splicing path is determined based on the remaining obstacle projections and the fusion area.

[0029] According to an embodiment of the present application, determining the target stitching path based on the remaining obstacle projection and the fusion area includes:

[0030] The initial stitching path is adjusted based on the remaining obstacle projections and the fusion area.

[0031] According to an embodiment of the present application, adjusting the initial stitching path based on the remaining obstacle projection and the fusion area includes:

[0032] In a case where it is determined that at least one of the remaining obstacle projections intersects the initial stitching path, adjusting the angle of the initial stitching path;

[0033] In a case where the adjusted initial stitching path does not intersect with any of the remaining obstacle projections, the adjusted initial stitching path is determined as the target stitching path.

[0034] According to an embodiment of the present application, adjusting the initial stitching path based on the remaining obstacle projection and the fusion area includes:

[0035] In a case where it is determined that at least one of the remaining obstacle projections intersects the initial stitching path, adjusting the angle of the initial stitching path;

[0036] When each adjusted initial stitching path intersects with at least one of the remaining obstacle projections, a path search algorithm is used to obtain at least one candidate stitching path corresponding to the fusion area;

[0037] In a case where a target candidate stitching path in the at least one candidate stitching path does not intersect with the remaining obstacle projections, the target candidate stitching path is determined as the target stitching path.

[0038] According to an embodiment of the present application, determining the target splicing path based on the position information of the obstacle projection and the position information of the predicted driving path includes:

[0039] In a case where all the obstacle projections intersect with the predicted driving path, the target splicing path is determined based on at least one of a priority corresponding to each obstacle projection and a distance between each obstacle projection and the target vehicle.

[0040] According to one embodiment of the present application, the driving parameters of the target vehicle include: at least one of a steering angle of the target vehicle and a driving direction of the vehicle.

[0041] According to an embodiment of the present application, generating an initial splicing path based on the fusion area includes:

[0042] A straight line stitching path generated based on the fusion area is determined as the initial stitching path, wherein the straight line stitching path is located in the fusion area, and the straight line stitching path has only a unique intersection with any straight line parallel to the two side boundary directions of the fusion area.

[0043] In a second aspect, the present application provides a panoramic surround image generation device, the device comprising:

[0044] A first processing module, configured to determine, based on a plurality of environment images of the target vehicle at a plurality of viewing angles, obstacle projection information and a fusion region between adjacent environment images of the target in the plurality of environment images;

[0045] A second processing module, used for generating an initial splicing path based on the fusion area;

[0046] A third processing module, configured to determine a target stitching path based on the initial stitching path and the obstacle projection information;

[0047] The fourth processing module is used to process the target adjacent environment image based on the target stitching path to obtain a panoramic surround image.

[0048] According to the panoramic surround view image generation device of the present application, by combining the initial stitching path generated based on the fusion area with the obstacle projection information to determine the target stitching path corresponding to the target adjacent environment image, it is possible to more flexibly avoid multiple obstacle projections to obtain a stitching path, and has a better blind spot elimination effect, thereby helping to improve the user experience and ensure driving safety; and it has a wide range of applicable scenarios.

[0049] In a third aspect, the present application provides a vehicle, comprising:

[0050] Image sensor;

[0051] The vehicle generates a panoramic surround view image based on the panoramic surround view image generation method as described in the first aspect.

[0052] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for generating a panoramic surround view image as described in the first aspect above is implemented.

[0053] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the panoramic surround image generation method as described in the first aspect above.

[0054] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0055] By combining the initial stitching path generated based on the fusion area with the obstacle projection information to determine the target stitching path corresponding to the target adjacent environment image, it is possible to more flexibly avoid multiple obstacle projections to obtain a stitching path, which has a better effect of eliminating blind spots, thereby helping to improve the user experience and ensure driving safety; and it has a wide range of applicable scenarios.

[0056] Furthermore, by combining the driving parameters of the target vehicle to search for a splicing path, when the obstacle projection cannot be avoided, a target splicing path that can prioritize the smaller obstacle blind spot in the driving trajectory is obtained for splicing, thereby ensuring that the obstacle blind spot in the driving trajectory is smaller, having a better blind spot elimination effect, and further improving robustness and driving safety.

[0057] Furthermore, in scenarios where obstacle boundaries are irregular or there are multiple obstacles, irregular splicing paths can be generated to flexibly avoid obstacle projections, thereby better adapting to scenarios where multiple obstacles exist.

[0058] Furthermore, in scenarios where blind spots cannot be eliminated, a spliced ​​path search is performed based on multiple dimensions such as the target vehicle's steering wheel angle, its speed, its gear position, the distance between the target vehicle and the obstacle projection, and the type of obstacle. This allows obstacles within the vehicle's driving trajectory or closest to the vehicle body, or obstacles with a higher priority level, to be fully displayed, further improving robustness and driving safety.

[0059] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0061] Figure 1 This is one of the flowcharts of the method for generating a panoramic surround image provided by an embodiment of the present application;

[0062] Figure 2 This is the second flow chart of the panoramic surround image generation method provided in the embodiment of the present application;

[0063] Figure 3 This is one of the principle schematic diagrams of the panoramic surround image generation method provided in the embodiment of the present application;

[0064] Figure 4 This is the second principle schematic diagram of the panoramic surround image generation method provided in the embodiment of the present application;

[0065] Figure 5 This is the third principle schematic diagram of the panoramic surround image generation method provided in the embodiment of the present application;

[0066] Figure 6 This is the fourth principle schematic diagram of the panoramic surround image generation method provided in the embodiment of the present application;

[0067] Figure 7 This is the fifth principle schematic diagram of the panoramic surround image generation method provided in the embodiment of the present application;

[0068] Figure 8 This is the sixth principle schematic diagram of the panoramic surround image generation method provided in the embodiment of the present application;

[0069] Fig. 9 It is one of the structural schematic diagrams of the panoramic surround image generation device provided in the embodiment of the present application;

[0070] Fig.10 This is the second structural schematic diagram of the panoramic surround image generation device provided in the embodiment of the present application;

[0071] Fig.11 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0073] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0074] In the following, in conjunction with the accompanying drawings, the panoramic surround image generation method, panoramic surround image generation device, electronic device and readable storage medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0075] The panoramic surround image generation method may be applied to a terminal, and may be specifically executed by hardware or software in the terminal.

[0076] The panoramic surround view image generation method provided in the embodiment of the present application may be executed by a vehicle or a functional module or functional entity in the vehicle that can implement the panoramic surround view image generation method, or may also be an electronic device or server that is communicatively connected to the vehicle. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The panoramic surround view image generation method provided in the embodiment of the present application is described below using a vehicle as an example of the execution subject.

[0077] like Figure 1 As shown, the panoramic surround image generation method includes: step 110, step 120, step 130 and step 140.

[0078] Step 110: determining the obstacle projection information and the fusion area between the target adjacent environment images in the multiple environment images based on the multiple environment images of the target vehicle at multiple viewing angles;

[0079] In this step, the environmental image may be environmental information around the target vehicle collected by image sensors disposed at different positions of the target vehicle, for example, it may be an environmental image perceived by the vehicle in the parking space when parking.

[0080] The target adjacent environment images may be any two adjacent environment images among the multiple environment images, for example, the environment image corresponding to the front and the environment image corresponding to the left; or the environment image corresponding to the front and the environment image corresponding to the right, etc.

[0081] The obstacle projection information is obstacle information, which is used to characterize the projection of obstacles around the target vehicle onto the ground plane.

[0082] The obstacle projection information includes: obstacle projection, the number of obstacle projections, and the intersection between obstacle projections, etc., wherein the obstacle projection is a projection image of the obstacle on the ground plane.

[0083] The number of obstacle projections may include 0, 1, or more.

[0084] It can be understood that when the number of obstacle projections is 0, it can be approximately considered that there is no obstacle in the environment image.

[0085] Figures 4 to 6 Various obstacle projection scenarios are illustrated, among which: Figure 4 This example shows a single obstacle projection scenario. Figure 5 and Figure 6 The following examples illustrate the scenes of multiple obstacle projections. Figure 5 The projections of obstacles in do not intersect each other. Figure 6 There are intersections in the obstacle projections.

[0086] In some embodiments, obstacles may include, but are not limited to, people, animals, other vehicles, and buildings.

[0087] The fusion area can be the overlapping area between adjacent environment image projections after the environment image is projected onto the ground plane, such as Figure 3 shown.

[0088] For example, in the actual implementation process, the fisheye camera can be used to collect environmental images corresponding to the front, rear, left and right sides of the target vehicle; obstacle detection can be performed on the four environmental images to obtain information such as the category and location of the obstacles in each environmental image.

[0089] The four environmental images and the position coordinates of the obstacles are inversely transformed and projected onto the ground plane to obtain the environmental image projection and obstacle projection information.

[0090] Step 120: Generate an initial splicing path based on the fusion area;

[0091] In this step, the initial stitching path can be obtained by the general panoramic imaging system (Around View Monitor) AVM stitching algorithm.

[0092] AVM is a panoramic imaging system that uses multiple ultra-wide-angle fisheye lenses to capture images and, through data processing and image correction technology, forms a panoramic image around the vehicle.

[0093] In some embodiments, step 120 may include:

[0094] The straight line stitching path generated based on the fusion area is determined as the initial stitching path.

[0095] In this embodiment, the initial splicing path is a straight line path.

[0096] The straight line stitching path is located in the fusion area, and the straight line stitching path has only one intersection with any straight line parallel to the boundary directions of both sides of the fusion area.

[0097] Step 130: determining a target stitching path based on the initial stitching path and obstacle projection information;

[0098] In this step, the target stitching path is a stitching path determined in the fusion area, which is used to stitch and fuse the target adjacent environment images.

[0099] It can be understood that for any two adjacent frames of environment images, a target splicing path can be correspondingly determined.

[0100] The target stitching paths determined by different adjacent environment images may be different, but the methods of determining the target stitching paths by different adjacent environment images are similar. The following will take the target adjacent environment images including the environment image corresponding to the front and the environment image corresponding to the left of the target vehicle as an example for explanation.

[0101] It should be noted that the target splicing path may be a straight line path, or may be an irregular curved path.

[0102] The target stitching path is a stitching path generated in the fusion area that can reduce the loss of obstacle information.

[0103] It can be understood that adjusting the initial obstacle stitching path according to the obstacle projection information can be manifested as allowing the stitching path to bypass all obstacles, or bypass a larger number of obstacles, or bypass obstacles with higher priorities, so that the image stitched based on the stitching path can retain more obstacles, so that the obstacle information loss is small to retain all obstacle information, or retain a larger number of obstacle information, or retain obstacle information with higher priority, etc.

[0104] After the initial stitching path is obtained, the number of obstacle projections and the intersection of the initial stitching path and the obstacle projections are determined.

[0105] In some embodiments, when it is determined based on the obstacle projection information that no obstacle projection intersects the initial stitching path, the initial stitching path is determined as the target stitching path.

[0106] like Figure 2As shown, for example, when the number of obstacle projections determined based on the obstacle projection information is 0; or when the number of obstacle projections determined based on the obstacle projection information is not 0, and the initial stitching path does not intersect with each obstacle projection, the initial stitching path can be used as the final target stitching path.

[0107] When it is determined based on the obstacle projection information that at least one obstacle projection intersects with the initial stitching path, the obstacle projection information and information such as the driving parameters of the target vehicle can be combined to adjust the initial stitching path or generate a new stitching path to obtain the target stitching path.

[0108] The driving parameters are the driving parameters of the target vehicle at the acquisition time corresponding to the current environment image.

[0109] In the actual implementation process, the target stitching path can be determined by combining the number of obstacle projections and the overlap between the obstacle projections and the fusion area.

[0110] Alternatively, based on the overlap between the obstacle projection and the fusion area, the target splicing path can be determined by predicting the possible intersection between the target vehicle and the obstacle projection based on the target vehicle's driving parameters.

[0111] In this step, the corresponding stitching path optimization method is selected according to the number of obstacle projections and the intersection of the obstacle projections with the initial stitching path. The adaptive stitching path optimization method can be flexibly selected based on different scenarios to avoid multiple obstacle projections as much as possible, improve the blind spot elimination effect, and quickly and accurately determine the target stitching path.

[0112] The implementation of step 130 is described in detail below.

[0113] like Figure 2 As shown, in some embodiments, step 130 may include:

[0114] When it is determined based on the obstacle projection information that at least one obstacle projection intersects the initial splicing path, a target splicing path is determined based on the driving parameters of the target vehicle.

[0115] In this embodiment, the driving parameters of the target vehicle are the driving parameters of the target vehicle at the acquisition time corresponding to the current environment image.

[0116] In some embodiments, the driving parameters of the target vehicle may include at least one of a steering angle and a driving direction of the target vehicle.

[0117] In this embodiment, the vehicle driving direction is used to represent the driving direction of the target vehicle in the following period of time, such as keeping straight, turning left, turning right, making a U-turn, reversing, and stopping.

[0118] The steering angle is used to characterize the change in the target vehicle's driving direction during the steering process compared to the previous time. When the steering angle is 0 or fluctuates around 0, it can be approximately considered that the target vehicle is moving straight or stationary.

[0119] Of course, in some embodiments, the driving parameters may also include: a steering wheel angle of the target vehicle, a wheel steering angle, a speed of the target vehicle, and a gear position of the target vehicle, etc.

[0120] It is understandable that, in actual implementation, the search for the stitching path may be performed based on the projections of each obstacle, such as by adjusting the angle of the initial stitching path or generating an irregular candidate stitching path through a path search algorithm.

[0121] Path search algorithms may include: breadth-first search algorithm and depth-first search algorithm, etc.

[0122] The irregular candidate stitching paths include curved paths. By setting the curved paths, irregular obstacle boundaries and a large number of obstacles can be avoided more flexibly than straight paths, thereby bypassing as many obstacles as possible, so that the stitched image retains more obstacle information.

[0123] When a stitching path that does not intersect with the projections of each obstacle can be found in the fusion area, the stitching path can be directly used as the target stitching path.

[0124] When a splicing path that does not intersect with any obstacle projection cannot be found, the splicing path search can be performed in combination with the driving parameters of the target vehicle so that the splicing path can avoid most obstacle projections to the greatest extent, or give priority to avoiding obstacle projections with higher obstacle priorities.

[0125] The target splicing path determined based on the projections of the obstacles may be a straight path or a curved path.

[0126] In this application, by considering different scenarios such as the number of obstacles and whether blind spots can be eliminated, a target splicing path matching the current scenario is obtained. It is suitable for various scenarios such as no obstacles, single obstacles, multiple obstacles, and obstacles that cannot be completely avoided. It has a wide range of application scenarios and high flexibility.

[0127] According to the panoramic surround view image generation method provided in the embodiment of the present application, a stitching path search is performed in combination with the driving parameters of the target vehicle, so that when the obstacle projection cannot be avoided, a target stitching path is obtained that can prioritize the smaller obstacle blind spots within the driving trajectory for stitching, thereby ensuring that the obstacle blind spots within the driving trajectory are smaller, having a better blind spot elimination effect, and further improving robustness and driving safety.

[0128] Continue to refer Figure 2 In some embodiments, determining a target splicing path based on the driving parameters of the target vehicle may include:

[0129] Based on the driving parameters, determining a predicted driving path corresponding to the target vehicle;

[0130] Based on the intersection of the predicted driving path and the fusion area, the target splicing path is determined.

[0131] In this embodiment, the predicted driving path is the predicted driving path of the target vehicle within a subsequent period of time.

[0132] like Figure 7 As shown, the predicted driving path may include multiple trajectory lines of the target vehicle, and a driving area is formed between the multiple trajectory lines.

[0133] In some embodiments, the predicted driving path of the target vehicle may be calculated based on the steering wheel angle of the target vehicle, the speed of the target vehicle, the gear position of the target vehicle, and the like.

[0134] The intersection situations include: intersection or non-intersection.

[0135] It is determined whether the predicted driving path passes through the fusion area, so as to select a corresponding method for generating the target splicing path based on whether the predicted driving path intersects with the fusion area.

[0136] The predicted driving path and the fusion area do not intersect, that is, the predicted driving path does not overlap with any part of the entire fusion area.

[0137] In this embodiment, when it is impossible to search for a stitching path that avoids all obstacle projections, the driving parameter search can prioritize the stitching path with a smaller obstacle blind spot within the driving trajectory. On this basis, panoramic stitching is performed based on the searched target stitching path, so that the panoramic surround view image finally stitched has a better blind spot elimination effect, which is suitable for scenes with multiple obstacles and scenes where blind spots cannot be eliminated.

[0138] According to the panoramic surround view image generation method provided in the embodiment of the present application, a stitching path search is performed in combination with the driving parameters of the target vehicle, so that when the obstacle projection cannot be avoided, a target stitching path is obtained that can prioritize the smaller obstacle blind spots within the driving trajectory for stitching, thereby ensuring that the obstacle blind spots within the driving trajectory are smaller, having a better blind spot elimination effect, and further improving robustness and driving safety.

[0139] Continue to refer Figure 2 In some embodiments, determining the target splicing path based on the intersection of the predicted driving path and the fusion area may include:

[0140] In the case that the predicted driving path and the fusion area do not intersect, a target splicing path is determined based on at least one of a priority corresponding to each obstacle projection and a distance between each obstacle projection and the target vehicle.

[0141] In this embodiment, the priority level corresponding to the obstacle projection is the priority level that needs to be retained during the stitching process.

[0142] The priority corresponding to the obstacle projection can be based on user customization or can be determined based on the category of the obstacle.

[0143] For example, when the obstacle projections include: projections corresponding to pedestrians and projections corresponding to stones, the priority of the projections corresponding to pedestrians may be higher than that corresponding to stones, that is, when performing image fusion, the projections corresponding to pedestrians should be retained first.

[0144] The distance between the obstacle projection and the target vehicle may be a pixel distance on the projection surface, or may be an actual physical distance between the obstacle and the target vehicle.

[0145] In some embodiments, the minimum rectangular frame enclosing the projection of the target vehicle on the ground plane is used as a reference, such as Figure 8 The rectangular frame (1) shown in FIG. 1 is continuously expanded outward with a fixed aspect ratio until the first intersection with the obstacle projection is generated. The intersection position is recorded as P1. The rectangular frame where the intersection P1 is located is as follows: Figure 8 The rectangular frame (2) shown;

[0146] The position of the projection of the center point of the rear axle of the target vehicle on the ground plane is recorded as P0, and the pixel distance between point P0 and point P1 is calculated to obtain the distance between the obstacle projection and the target vehicle.

[0147] In some embodiments, the pixel distance may also be converted to an actual physical distance.

[0148] In some embodiments, the pixel distances between each pixel point in the projection of the target vehicle on the ground plane and each pixel point in the projection of the obstacle may also be calculated, and the pixel distance with the shortest pixel distance may be used as the distance between the obstacle projection and the target vehicle.

[0149] Of course, in other embodiments, other methods for calculating the distance may also be used, and this application does not limit this.

[0150] In some embodiments, after calculating the distance between each obstacle projection and the target vehicle, the target stitching path can be generated with the goal of avoiding one or more obstacle projections that are closest to the target vehicle. That is, when performing image fusion, the projections corresponding to the previous one or a certain number of obstacles that are closest to the target vehicle are retained first.

[0151] In the actual execution process, when the predicted driving path does not pass through the fusion area, it can be approximately considered that the target vehicle is less likely to collide with obstacles in the fusion area in the future. In this case, a target stitching path can be generated based on the priority corresponding to each obstacle projection in the fusion area and the distance between each obstacle projection and the target vehicle, so that the stitched image can effectively retain the obstacle projections with higher priority or the obstacle projections that are closer to the target vehicle.

[0152] For example, after obtaining the priority corresponding to each obstacle projection and the distance between each obstacle projection and the target vehicle, the obstacle projections can be sorted in ascending order according to the distance to obtain the obstacle projection with the closest distance; if there are multiple shortest distances, the obstacle projections corresponding to the multiple shortest distances can be further sorted from important to secondary according to the category of the obstacle corresponding to the obstacle projection, so as to screen out the most important obstacle projection, that is, the obstacle projection with the highest priority.

[0153] In some embodiments, determining the target splicing path based on at least one of the priority corresponding to each obstacle projection and the distance between each obstacle projection and the target vehicle may further include:

[0154] Perform weighted calculation on the priority corresponding to each obstacle projection and the distance between each obstacle projection and the target vehicle to obtain the score corresponding to each obstacle projection;

[0155] Based on the obstacle projection with the highest scoring target number, the target stitching path is determined.

[0156] In this embodiment, the target quantity can be customized based on the user, such as being set to 1 or 3.

[0157] The higher the score, the more likely the obstacle projection corresponding to the score should be retained during the fusion process. That is, in the process of generating a stitching path, the generated stitching path should be made to avoid the obstacle projection corresponding to the score first.

[0158] According to the panoramic surround view image generation method provided in the embodiment of the present application, in a scenario where the blind spot cannot be eliminated, a stitching path search is performed based on multiple dimensions such as the target vehicle's steering wheel angle, the vehicle's speed, the vehicle's gear position, the distance between the target vehicle and the obstacle projection, and the type of obstacle. Priority is given to ensuring that obstacles within the vehicle's driving trajectory or closest to the vehicle body, or obstacles with a higher priority level are fully displayed, thereby further improving robustness and driving safety.

[0159] Continue to refer Figure 2 In some embodiments, determining the target splicing path based on the intersection of the predicted driving path and the fusion area may include:

[0160] When the predicted driving path and the fusion area intersect, the target splicing path is determined based on the position information of the obstacle projection and the position information of the predicted driving path.

[0161] In this embodiment, the position information of the obstacle projection is used to represent the position of the obstacle projection on the projection surface where the obstacle projection is located.

[0162] The position information of the predicted driving path is used to characterize the position of the predicted driving path on the projection surface where the obstacle projection is located.

[0163] In some embodiments, the location information may be determined based on the following steps:

[0164] Obtaining a set of pixel regions of each obstacle projection and parameter equations of at least two trajectory lines corresponding to the predicted driving path;

[0165] Based on the parameter equations of at least two trajectory lines, a driving area pixel set in a driving area formed between the at least two trajectory lines is obtained;

[0166] The position information of the predicted driving path is determined based on the driving area pixel set; and the position information of the obstacle projection is determined based on the obstacle projection pixel area set.

[0167] In this embodiment, the predicted driving path may include multiple trajectory lines of the target vehicle, and a driving area is formed between the multiple trajectory lines.

[0168] The driving area pixel set is a pixel position set of pixel points of the driving area on the projection surface where the obstacle projection is located.

[0169] The pixel area set of the obstacle projection is a pixel position set of pixel points of the obstacle projection on the projection surface where the obstacle projection is located.

[0170] In some embodiments, determining the target splicing path based on the position information of the obstacle projection and the position information of the predicted driving path may include:

[0171] Based on the position information of the obstacle projection and the position information of the predicted driving path, determining the intersection of the obstacle projection and the predicted driving path;

[0172] The target splicing path is determined according to the intersection of the obstacle projection and the predicted driving path.

[0173] In this embodiment, based on the position information of the obstacle projection and the position information of the predicted driving path, it can be determined whether the predicted driving path intersects with each obstacle projection.

[0174] After obtaining the pixel area set of each obstacle projection and the pixel set of the driving area, traverse the pixel area set of each obstacle projection and perform an intersection operation on it with the pixel set of the driving area;

[0175] If the result is an empty set, it means that the obstacle projection does not intersect with the predicted driving path;

[0176] If the result is not empty, it means that the obstacle projection intersects with the predicted driving path.

[0177] For example, if the position information of the predicted driving path partially overlaps with the position information of the obstacle projection, it can be considered that the predicted driving path and the obstacle projection at least partially intersect; for example, if the position information of the predicted driving path and the position information of the obstacle projection are different, it can be approximately considered that the predicted driving path and the obstacle projection do not intersect.

[0178] Continue to refer Figure 2 In some embodiments, determining the target splicing path based on the position information of the obstacle projections and the position information of the predicted driving path may include: when all obstacle projections intersect with the predicted driving path, determining the target splicing path based on at least one of the priorities corresponding to the obstacle projections and the distances between the obstacle projections and the target vehicle.

[0179] In this embodiment, when the predicted driving path passes through the fusion area, Figure 6As shown in the figure, it can be approximately considered that the target vehicle is more likely to collide with the obstacles in the fusion area in the future. In this case, it is prioritized to determine whether there is a splicing path that can avoid all obstacle projections, so that the spliced ​​image can retain all obstacle projections; when there is no splicing path that can avoid all obstacle projections, a splicing path is generated in combination with the priority corresponding to each obstacle projection and the distance between each obstacle projection and the target vehicle, so as to prioritize ensuring that the obstacle blind spot in the predicted driving path is small when the obstacle cannot be avoided.

[0180] The specific method for determining the target splicing path based on at least one of the priorities corresponding to the obstacle projections and the distances between the obstacle projections and the target vehicle has been described in the above embodiments and will not be elaborated in this application.

[0181] Continue to refer Figure 2 In some embodiments, determining the target splicing path based on the position information of the obstacle projection and the position information of the predicted driving path may include:

[0182] When at least one obstacle projection among all obstacle projections does not intersect with the predicted driving path, the obstacle projection that does not intersect with the predicted driving path is deleted;

[0183] Based on the projection information of the deleted obstacles, the target stitching path is determined.

[0184] In this embodiment, the deleted obstacle projection information is used to characterize the number of remaining obstacle projections after deleting the obstacle projections that do not intersect with the predicted driving path, and the intersection of the remaining obstacle projections with the fusion area when there are remaining obstacle projections.

[0185] It can be understood that at least one obstacle projection among all obstacle projections does not intersect with the predicted driving path, which may include: all obstacle projections do not intersect with the predicted driving path; and some obstacle projections do not intersect with the predicted driving path, and the other part of the obstacle projections intersect with the predicted driving path.

[0186] Based on the above different situations, different target splicing path determination methods can be selected.

[0187] In some embodiments, determining the target stitching path based on the deleted obstacle projection information may include:

[0188] When it is determined that no remaining obstacle projections exist based on the deleted obstacle projection information, a target stitching path is determined based on the fusion area.

[0189] In this embodiment, in the case where all obstacle projections do not intersect with the predicted driving path, after deleting the obstacle projections that do not intersect with the predicted driving path, there are no remaining obstacle projections.

[0190] In this case, it is only necessary to generate a target stitching path based on the fusion area, such as generating a straight stitching path using a method similar to the initial stitching path, and using the generated straight stitching path as the target stitching path.

[0191] Continue to refer Figure 2 In some embodiments, determining the target stitching path based on the deleted obstacle projection information may include:

[0192] In the case where it is determined that there are remaining obstacle projections based on the deleted obstacle projection information, a target splicing path is determined based on the remaining obstacle projections and the fusion area.

[0193] In this embodiment, the remaining obstacle projections are obstacle projections that intersect with the predicted driving path.

[0194] Step 130 may be repeatedly performed based on the obstacle projections and fusion areas intersecting the predicted driving path to determine a target spliced ​​path.

[0195] For example, the intersection of the retained obstacle projections and the initial stitching path can be re-determined. When at least one of the retained obstacle projections intersects with the initial stitching path, the target stitching path can be re-determined by adjusting the angle of the initial stitching path, using a path search algorithm to obtain at least one candidate stitching path corresponding to the fusion area, or combining the driving parameters of the target vehicle.

[0196] In some embodiments, determining a target stitching path based on the remaining obstacle projections and the fusion area may further include:

[0197] Based on the remaining obstacle projections and fusion areas, adjust the initial stitching path.

[0198] In this embodiment, adjusting the initial splicing path is performed as dynamic splicing.

[0199] Adjusting the initial splicing path may include, but is not limited to: adjusting the position of the initial splicing path and adjusting the line shape of the initial splicing path.

[0200] The initial stitching path is adjusted to select a path that exists within the fusion area and does not intersect with the remaining obstacle projections.

[0201] The following describes how to adjust the initial stitching path.

[0202] In some embodiments, adjusting the initial stitching path based on the remaining obstacle projections and the fusion area may include:

[0203] In the case where it is determined that there is at least one remaining obstacle projection intersecting the initial stitching path, adjusting the angle of the initial stitching path;

[0204] When the adjusted initial stitching path does not intersect with any remaining obstacle projections, the adjusted initial stitching path is determined as the target stitching path.

[0205] In this embodiment, the obstacle projection intersects the initial stitching path, which may be manifested as at least a portion of the obstacle projection overlapping at least a portion of the initial stitching path, such as the initial stitching path passing through the obstacle projection, or the initial stitching path being within the obstacle projection.

[0206] The obstacle projection does not intersect with the initial stitching path, that is, there is no overlap between the initial stitching path and any part of the entire obstacle projection.

[0207] When the initial stitching path intersects with the obstacle projection, the angle of the initial stitching path can be adjusted in a clockwise or counterclockwise direction to move the initial stitching path toward the fusion area until a stitching path that does not intersect with the obstacle projections in the fusion area is obtained, and the stitching path is determined as the target stitching path, such as Figure 4 shown.

[0208] When a stitching path that does not intersect with any obstacle projection cannot be found by adjusting the angle of the initial stitching path, for example, when there are a large number of obstacle projections, the obstacle projections are irregular, or there are intersecting obstacle projections, an irregular stitching path can be generated to avoid the obstacle projections as much as possible.

[0209] According to the panoramic surround image generation method provided by the embodiment of the present application, it is possible to quickly obtain a target stitching path that avoids obstacle projections by adjusting the angle of the initial stitching path in a single obstacle scene.

[0210] In some embodiments, adjusting the initial stitching path based on the remaining obstacle projections and the fusion area may include:

[0211] In the case where it is determined that there is at least one remaining obstacle projection intersecting the initial stitching path, adjusting the angle of the initial stitching path;

[0212] When each adjusted initial stitching path intersects with at least one remaining obstacle projection, a path search algorithm is used to obtain at least one candidate stitching path corresponding to the fusion area;

[0213] When the target candidate stitching path in at least one candidate stitching path does not intersect with any of the remaining obstacle projections, the target candidate stitching path is determined as the target stitching path.

[0214] In this embodiment, the angle of the initial stitching path may be adjusted based on a clockwise or counterclockwise direction so that the initial stitching path moves toward the fusion area.

[0215] Path search algorithms may include: breadth-first search algorithm and depth-first search algorithm, etc.

[0216] The candidate splicing path obtained by searching based on the path search algorithm may be an irregular dividing line.

[0217] Continue to refer Figure 2 , when it is impossible to find a stitching path that does not intersect with the projections of each obstacle in the fusion area by adjusting the angle of the initial stitching path, a path search algorithm can be used to find an irregular candidate stitching path that satisfies the condition of not meeting the projections of each obstacle, thereby obtaining the target stitching path, such as Figure 5 shown.

[0218] When the path search algorithm still cannot find a candidate splicing path that does not intersect with any obstacle projection, for example, when there are a large number of obstacle projections, the search can be combined with the driving parameters of the target vehicle, and priority can be given to ensuring that the obstacle blind spot within the driving trajectory is small, so as to achieve the purpose of visually ensuring driving safety for users.

[0219] According to the panoramic surround image generation method provided in the embodiment of the present application, it is possible to flexibly avoid obstacle projections by generating irregular stitching paths in scenarios where obstacle boundaries are irregular or there are multiple obstacles, thereby better adapting to scenarios where multiple obstacles exist and further improving the effect of eliminating blind spots.

[0220] Continue to refer Figure 2 In some embodiments, step 130 may further include:

[0221] First, adjust the initial stitching path based on obstacle projection and fusion area;

[0222] In the case that there is no splicing path that does not intersect with the projections of each obstacle, a target splicing path is determined based on the driving parameters of the target vehicle.

[0223] In this embodiment, the specific implementation method of adjusting the initial stitching path based on the obstacle projection and the fusion area has been described in the above embodiment, and will not be repeated here.

[0224] For example, the target stitching path can be first found by adjusting the angle or using path search algorithms. When there is no stitching path that does not intersect with the projections of all obstacles, the target stitching path can be determined based on the intersection of the predicted driving path and the fusion area in combination with the driving parameters of the target vehicle.

[0225] Among them, the specific implementation method of determining the target splicing path based on the intersection of the predicted driving path and the fusion area has been described in the above embodiment, and this application will not repeat it here.

[0226] According to the panoramic surround image generation method provided in the embodiment of the present application, by providing a variety of stitching path adjustment algorithms, it is possible to flexibly select based on different scenarios such as the number of obstacles and whether blind spots can be eliminated, and obtain a target stitching path that matches the current scene. It is suitable for a variety of scenarios such as no obstacles, a single obstacle, multiple obstacles, and obstacles that cannot be completely avoided, and has a wide range of application scenarios and high flexibility.

[0227] Step 140: Process the target adjacent environment image based on the target stitching path to obtain a panoramic surround image.

[0228] In this step, the Around View Monitor (AVM) is a panoramic image obtained by dedistorting and inverse perspective transforming four fisheye images and stitching them together to facilitate users to observe the situation around the vehicle.

[0229] Through steps 110 to 130, target stitching paths corresponding to any two adjacent environment images can be obtained respectively.

[0230] After the target stitching paths are obtained, based on the target stitching paths, the projection images of the target adjacent environment images corresponding to the target stitching paths on the ground plane are stitched and fused respectively to obtain a panoramic surround image.

[0231] In this application, an initial stitching path is generated by fusion area, and the target stitching path is obtained by further optimizing the stitching path by combining the initial stitching path and obstacle projection information. Factors such as the number of obstacles can be considered to obtain a target stitching path that matches the current scene. It is suitable for various scenes such as no obstacles, single obstacles, and multiple obstacles, and has a wide range of application scenarios and high flexibility.

[0232] In addition, by generating a target stitching path for image stitching, compared with the related art which uses an image segmentation-based method to screen the segmented sub-images to fuse the screened sub-images to achieve image stitching, the present application can utilize the flexible and changeable characteristics of the stitching path, has higher flexibility and obstacle avoidance accuracy, and further improves the stitching effect.

[0233] According to the panoramic surround view image generation method provided in the embodiment of the present application, by combining the initial stitching path generated based on the fusion area and the obstacle projection information to determine the target stitching path corresponding to the target adjacent environment image, it is possible to more flexibly avoid multiple obstacle projections to obtain a stitching path, and has a better blind spot elimination effect, thereby helping to improve the user experience and ensure driving safety; and it has a wide range of applicable scenarios.

[0234] In some embodiments, the Fig. 9 The AVM synthesis module shown is used to execute the steps of the method of the present application.

[0235] Among them, the input end of the AVM synthesis module is used to input multiple environmental images and driving parameters of the target vehicle, so as to determine the target stitching path based on the input multiple environmental images and driving parameters of the target vehicle, and process the target adjacent environmental images and panoramic surround images based on the target stitching path.

[0236] The panoramic surround image generation method provided in the embodiment of the present application can be executed by a panoramic surround image generation device. In the embodiment of the present application, the panoramic surround image generation method executed by the panoramic surround image generation device is taken as an example to illustrate the panoramic surround image generation device provided in the embodiment of the present application.

[0237] The embodiment of the present application also provides a panoramic surround image generation device.

[0238] like Fig.10 As shown, the panoramic surround image generation device includes: a first processing module 1010 , a second processing module 1020 , a third processing module 1030 and a fourth processing module 1040 .

[0239] A first processing module 1010 is used to determine the obstacle projection information and the fusion area between the target adjacent environment images in the multiple environment images based on the multiple environment images of the target vehicle at multiple perspectives;

[0240] The second processing module 1020 is used to generate an initial splicing path based on the fusion area;

[0241] The third processing module 1030 is used to determine a target stitching path based on the initial stitching path and the obstacle projection information;

[0242] The fourth processing module 1040 is used to process the target adjacent environment image based on the target stitching path to obtain a panoramic surround image.

[0243] According to the panoramic surround view image generation device provided in the embodiment of the present application, by combining the initial stitching path generated based on the fusion area and the obstacle projection information to determine the target stitching path corresponding to the target adjacent environment image, it is possible to more flexibly avoid multiple obstacle projections to obtain a stitching path, and has a better blind spot elimination effect, thereby helping to improve the user experience and ensure driving safety; and it has a wide range of applicable scenarios.

[0244] In some embodiments, the third processing module 1030 may also be used to:

[0245] Based on the initial stitching path and obstacle projection information, the target stitching path is determined, including:

[0246] When it is determined based on the obstacle projection information that at least one obstacle projection intersects the initial splicing path, a target splicing path is determined based on the driving parameters of the target vehicle.

[0247] In some embodiments, the third processing module 1030 may also be used to:

[0248] Based on the driving parameters, determining a predicted driving path corresponding to the target vehicle;

[0249] Based on the intersection of the predicted driving path and the fusion area, the target splicing path is determined.

[0250] In some embodiments, the third processing module 1030 may also be used to:

[0251] In the case that the predicted driving path and the fusion area do not intersect, a target splicing path is determined based on at least one of a priority corresponding to each obstacle projection and a distance between each obstacle projection and the target vehicle.

[0252] In some embodiments, the third processing module 1030 may also be used to:

[0253] When the predicted driving path and the fusion area intersect, the target splicing path is determined based on the position information of the obstacle projection and the position information of the predicted driving path.

[0254] In some embodiments, the third processing module 1030 may also be used to:

[0255] Based on the position information of the obstacle projection and the position information of the predicted driving path, determining the intersection of the obstacle projection and the predicted driving path;

[0256] The target splicing path is determined according to the intersection of the obstacle projection and the predicted driving path.

[0257] In some embodiments, the third processing module 1030 may also be used to:

[0258] When at least one obstacle projection among all obstacle projections does not intersect with the predicted driving path, the obstacle projection that does not intersect with the predicted driving path is deleted;

[0259] Based on the projection information of the deleted obstacles, the target stitching path is determined.

[0260] In some embodiments, the third processing module 1030 may also be used to:

[0261] When it is determined that no remaining obstacle projections exist based on the deleted obstacle projection information, a target stitching path is determined based on the fusion area.

[0262] In some embodiments, the third processing module 1030 may also be used to:

[0263] In the case where it is determined that there are remaining obstacle projections based on the deleted obstacle projection information, a target splicing path is determined based on the remaining obstacle projections and the fusion area.

[0264] In some embodiments, the third processing module 1030 may also be used to:

[0265] Based on the remaining obstacle projections and fusion areas, adjust the initial stitching path.

[0266] In some embodiments, the third processing module 1030 may also be used to:

[0267] In the case where it is determined that there is at least one remaining obstacle projection intersecting the initial stitching path, adjusting the angle of the initial stitching path;

[0268] When the adjusted initial stitching path does not intersect with any remaining obstacle projections, the adjusted initial stitching path is determined as the target stitching path.

[0269] In some embodiments, the third processing module 1030 may also be used to:

[0270] In the case where it is determined that there is at least one remaining obstacle projection intersecting the initial stitching path, adjusting the angle of the initial stitching path;

[0271] When each adjusted initial stitching path intersects with at least one remaining obstacle projection, a path search algorithm is used to obtain at least one candidate stitching path corresponding to the fusion area;

[0272] When the target candidate stitching path in at least one candidate stitching path does not intersect with any of the remaining obstacle projections, the target candidate stitching path is determined as the target stitching path.

[0273] In some embodiments, the third processing module 1030 may also be used to:

[0274] When all obstacle projections intersect with the predicted driving path, a target splicing path is determined based on at least one of a priority corresponding to each obstacle projection and a distance between each obstacle projection and the target vehicle.

[0275] In some embodiments, the second processing module 1020 may also be used to:

[0276] A straight line stitching path generated based on the fusion area is determined as an initial stitching path, wherein the straight line stitching path is located in the fusion area, and the straight line stitching path has only a unique intersection with any straight line parallel to the boundary directions of both sides of the fusion area.

[0277] The panoramic surround image generation device in the embodiment of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS) or a personal computer (personal computer, PC), etc., which is not specifically limited in the embodiment of the present application.

[0278] The panoramic surround image generation device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0279] The panoramic surround image generation device provided in the embodiment of the present application can achieve Figures 1 to 8 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0280] An embodiment of the present application also provides a vehicle.

[0281] The vehicle includes: an image sensor.

[0282] In this embodiment, the number of image sensors may be plural.

[0283] Multiple image sensors are used to collect multiple environmental images of the target vehicle at multiple viewing angles.

[0284] The vehicle generates a panoramic surround view image based on the panoramic surround view image generation method described in any of the above embodiments.

[0285] According to the vehicle provided in the embodiment of the present application, by combining the initial stitching path generated based on the fusion area and the obstacle projection information to determine the target stitching path corresponding to the target adjacent environment image, it is possible to more flexibly avoid multiple obstacle projections to obtain a stitching path, and has a better blind spot elimination effect, thereby helping to improve the user experience and ensure driving safety; and it has a wide range of applicable scenarios.

[0286] In some embodiments, Fig.11 As shown, an embodiment of the present application also provides an electronic device 1100, including a processor 1101, a memory 1102, and a computer program stored in the memory 1102 and executable on the processor 1101. When the program is executed by the processor 1101, each process of the above-mentioned panoramic surround image generation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0287] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0288] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned panoramic surround image generation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0289] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0290] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned panoramic surround image generation method when executed by a processor.

[0291] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0292] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned panoramic surround image generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0293] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0294] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0295] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0296] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0297] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0298] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A method for generating a panoramic surround image, characterized in that: include: Determining, based on a plurality of environment images of the target vehicle at a plurality of viewing angles, a fusion region between obstacle projection information and adjacent environment images of the target in the plurality of environment images; Based on the fusion region, generating an initial splicing path; Determining a target stitching path based on the initial stitching path and the obstacle projection information; Processing the target adjacent environment image based on the target stitching path to obtain a panoramic surround image; The determining a target stitching path based on the initial stitching path and the obstacle projection information includes: In a case where it is determined based on the obstacle projection information that at least one obstacle projection intersects with the initial splicing path, determining the target splicing path based on the driving parameters of the target vehicle; The step of determining the target splicing path based on the driving parameters of the target vehicle includes: Based on the driving parameters, determining a predicted driving path corresponding to the target vehicle; The target splicing path is determined based on the intersection of the predicted driving path and the fusion area.

2. The method for generating a panoramic surround image according to claim 1, characterized in that: The determining the target splicing path based on the intersection of the predicted driving path and the fusion area includes: In a case where the predicted driving path and the fusion area do not intersect, the target splicing path is determined based on at least one of a priority corresponding to each obstacle projection and a distance between each obstacle projection and the target vehicle.

3. The method for generating a panoramic surround image according to claim 1, characterized in that: The determining the target splicing path based on the intersection of the predicted driving path and the fusion area includes: In a case where the predicted driving path and the fusion area intersect, the target splicing path is determined based on the position information of the obstacle projection and the position information of the predicted driving path.

4. The method for generating a panoramic surround image according to claim 3, characterized in that: The determining the target splicing path based on the position information of the obstacle projection and the position information of the predicted driving path includes: Based on the position information of the obstacle projection and the position information of the predicted driving path, determining the intersection of the obstacle projection and the predicted driving path; The target splicing path is determined according to the intersection of the obstacle projection and the predicted driving path.

5. The method for generating a panoramic surround image according to claim 3, characterized in that: The determining the target splicing path based on the position information of the obstacle projection and the position information of the predicted driving path includes: When at least one obstacle projection among all the obstacle projections does not intersect with the predicted driving path, deleting the obstacle projection that does not intersect with the predicted driving path; The target splicing path is determined based on the deleted obstacle projection information.

6. The method for generating a panoramic surround image according to claim 5, characterized in that: The determining the target splicing path based on the deleted obstacle projection information includes: When it is determined based on the deleted obstacle projection information that there is no remaining obstacle projection, the target splicing path is determined based on the fusion area.

7. The method for generating a panoramic surround image according to claim 5, characterized in that: The determining the target splicing path based on the deleted obstacle projection information includes: In a case where it is determined based on the deleted obstacle projection information that there are remaining obstacle projections, the target splicing path is determined based on the remaining obstacle projections and the fusion area.

8. The method for generating a panoramic surround image according to claim 7, characterized in that: The determining the target stitching path based on the remaining obstacle projection and the fusion area includes: The initial stitching path is adjusted based on the remaining obstacle projections and the fusion area.

9. The method for generating a panoramic surround image according to claim 8, characterized in that: The adjusting the initial stitching path based on the remaining obstacle projection and the fusion area includes: In a case where it is determined that at least one of the remaining obstacle projections intersects the initial stitching path, adjusting the angle of the initial stitching path; In a case where the adjusted initial stitching path does not intersect with any of the remaining obstacle projections, the adjusted initial stitching path is determined as the target stitching path.

10. The method for generating a panoramic surround image according to claim 8, characterized in that: The adjusting the initial stitching path based on the remaining obstacle projection and the fusion area includes: In a case where it is determined that at least one of the remaining obstacle projections intersects the initial stitching path, adjusting the angle of the initial stitching path; When each adjusted initial stitching path intersects with at least one of the remaining obstacle projections, a path search algorithm is used to obtain at least one candidate stitching path corresponding to the fusion area; In a case where a target candidate stitching path in the at least one candidate stitching path does not intersect with the remaining obstacle projections, the target candidate stitching path is determined as the target stitching path.

11. The method for generating a panoramic surround image according to claim 3, characterized in that: The determining the target splicing path based on the position information of the obstacle projection and the position information of the predicted driving path includes: In a case where all the obstacle projections intersect with the predicted driving path, the target splicing path is determined based on at least one of a priority corresponding to each obstacle projection and a distance between each obstacle projection and the target vehicle.

12. The method for generating a panoramic surround image according to claim 1, characterized in that: The driving parameters of the target vehicle include at least one of a steering angle and a driving direction of the target vehicle.

13. The method for generating a panoramic surround image according to any one of claims 1 to 12, characterized in that: The generating an initial splicing path based on the fusion region includes: A straight line stitching path generated based on the fusion area is determined as the initial stitching path, wherein the straight line stitching path is located in the fusion area, and the straight line stitching path has only a unique intersection with any straight line parallel to the two side boundary directions of the fusion area.

14. A vehicle, characterized in that: include: Image sensor; The vehicle generates a panoramic surround view image based on the panoramic surround view image generation method as described in any one of claims 1 to 13.

15. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating a panoramic surround view image as described in any one of claims 1 to 13 is implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the panoramic surround image generation method as described in any one of claims 1 to 13 is implemented.

Images