Door obstacle avoidance method, electronic device, and computer-readable storage medium

Abstract

The present application discloses a door obstacle avoidance method, an electronic device, and a computer-readable storage medium. The method includes: obtaining a semantic segmentation image and a depth image corresponding to an image to be detected; wherein, the semantic segmentation image includes semantic categories corresponding to image regions, the depth image includes depth information corresponding to pixels, and the obstacles matching multiple semantic categories include step-like obstacles and non-step-like obstacles; determining a detection height corresponding to the step-like obstacle based on the coordinates matched by multiple pixels in the image region, and determining a detection height corresponding to the non-step-like obstacle based on the coordinates matched by the pixels at a specified position in the image region and the depth information included in the depth image; determining a target door opening parameter corresponding to the door based on the detection height corresponding to each type of obstacle and the reference height corresponding to the door; wherein, the target door opening parameter is used to avoid obstacles higher than the reference height when the door is opened. By the above method, the present application can improve the accuracy of door obstacle avoidance.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular, to a door obstacle avoidance method, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the development of vehicle intelligence, automatically opening and closing doors have become an important branch of vehicle intelligence. However, during the automatic opening and closing process, the doors are extremely likely to touch obstacles and get damaged. In the prior art, usually, the collected images are sent to the in-vehicle terminal for the user to confirm independently, or the distance between the door and the obstacle is detected by radar to determine whether the obstacle can be avoided. However, it is difficult for related technologies to accurately determine the specific height of different types of obstacles, resulting in low accuracy of door obstacle avoidance. In view of this, how to improve the accuracy of door obstacle avoidance has become an urgent problem to be solved. Summary of the Invention

[0003] The main technical problem to be solved by the present application is to provide a door obstacle avoidance method, an electronic device, and a computer-readable storage medium, which can improve the accuracy of door obstacle avoidance.

[0004] To solve the above technical problem, in a first aspect of the present application, a door obstacle avoidance method is provided, including: obtaining a semantic segmentation image and a depth image corresponding to an image to be detected; wherein, the semantic segmentation image includes semantic categories corresponding to image regions, the depth image includes depth information corresponding to pixels, and the obstacles matching multiple semantic categories include step-like obstacles and non-step-like obstacles; determining the detection height corresponding to the step-like obstacles based on the coordinates matched by multiple pixels in the image region, and determining the detection height corresponding to the non-step-like obstacles based on the coordinates matched by the pixels at a specified position in the image region and the depth information included in the depth image; determining the target opening parameter corresponding to the door based on the detection height corresponding to each type of obstacle and the reference height corresponding to the door; wherein, the target opening parameter is used to avoid obstacles higher than the reference height when the door is opened.

[0005] To solve the above technical problem, in a second aspect of the present application, an electronic device is provided, which includes: a memory and a processor coupled to each other, wherein, the memory stores program data, and the processor calls the program data to execute the method described in the first aspect above.

[0006] To solve the above technical problem, in a third aspect of the present application, a computer-readable storage medium is provided, on which program data is stored, and when the program data is executed by a processor, the method described in the first aspect above is implemented.

[0007] In the above solution, a to-be-detected image is obtained, and a semantic segmentation image and a depth image corresponding to the to-be-detected image are determined. Different semantic categories corresponding to different image regions can be determined from the semantic segmentation image, and the depth information corresponding to pixels can be obtained from the depth image. Multiple semantic categories correspond to different categories of obstacles, and the multiple categories of obstacles include step-like obstacles and non-step-like obstacles. Based on the coordinates matched by multiple pixels in the image region, the height of the step-like obstacles is analyzed to determine the detected height corresponding to the step-like obstacles. Thus, when the height drop of the step-like obstacles changes little and the depth information difference is not obvious, the detected height of the obstacles is determined by multi-point sampling. Based on the coordinates matched by the pixels at the specified position in the image region and the depth information included in the depth image, the height of the non-step-like obstacles is analyzed to determine the detected height corresponding to the non-step-like obstacles. Thus, when the height drop of the non-step-like obstacles changes greatly, the height of the non-step-like obstacles in the three-dimensional space is detected by combining the depth information, and finally the accuracy of the detected height corresponding to the obstacles with different height drops is improved. Based on the detected height corresponding to each type of obstacle and the reference height corresponding to the vehicle door, how to set the vehicle door parameters is analyzed to determine the target opening parameters corresponding to the vehicle door, so that when the vehicle door is opened according to the target opening parameters, obstacles higher than the reference height can be avoided, thereby reducing the probability of collision between the opened vehicle door and the obstacles and improving the accuracy of vehicle door obstacle avoidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. Among them:

[0009] Figure 1 is a schematic flowchart of an embodiment of the vehicle door obstacle avoidance method of the present application;

[0010] Figure 2 is a schematic flowchart of another embodiment of the vehicle door obstacle avoidance method of the present application;

[0011] Figure 3 is a schematic application scenario diagram of an embodiment of obtaining the detected height of step-like obstacles of the present application;

[0012] Figure 4 is a schematic structural diagram of an embodiment of an electronic device of the present application;

[0013] Figure 5 is a schematic structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0014] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all embodiments, and adaptive combinations can be made between different embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0015] The terms "system" and "network" are often used interchangeably in this document. The term "and / or" in this document only describes the association relationship of associated objects and indicates that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this document generally represents an "or" relationship between the associated objects before and after. In addition, "multiple" in this document means two or more than two.

[0016] The door obstacle avoidance method provided by the present application is used to set target door opening parameters for automatically opening and closing doors so as to avoid obstacles that may damage the doors, and its corresponding execution entity is a processing unit capable of image processing.

[0017] Please refer to Figure 1 , Figure 1 is a schematic flowchart of an implementation manner of the door obstacle avoidance method of the present application. The method includes:

[0018] S101: Obtain a semantic segmentation image and a depth image corresponding to the image to be detected; wherein, the semantic segmentation image includes the semantic categories corresponding to the image regions, the depth image includes the depth information corresponding to the pixels, and the obstacles matching multiple semantic categories include step-like obstacles and non-step-like obstacles.

[0019] Specifically, obtain the image to be detected, determine the semantic segmentation image and the depth image corresponding to the image to be detected, the semantic categories corresponding to different image regions can be determined from the semantic segmentation image, and the depth information corresponding to the pixels can be obtained from the depth image. Multiple semantic categories correspond to different categories of obstacles, and the multiple categories of obstacles include step-like obstacles and non-step-like obstacles.

[0020] In some implementation scenarios, obtain the image to be detected, use a semantic segmentation model to perform semantic segmentation on the pixels in the image to be detected to obtain a semantic segmentation image corresponding to the image to be detected, and use a depth detection model to perform depth analysis on the pixels in the image to be detected to obtain a depth image corresponding to the image to be detected. Among them, the semantic segmentation model is trained using images with semantic category labels, and the depth detection model is trained using images with depth information labels.

[0021] In some implementation scenarios, a to-be-detected image is obtained, and a multi-task detection model is used to determine the semantic category and depth information matched by the pixels in the to-be-detected image. Among them, the multi-task detection model first performs semantic segmentation on the pixels in the to-be-detected image, and then performs depth analysis on the pixels belonging to different semantic categories based on the result of the semantic segmentation. The multi-task detection model is trained using images with semantic category labels and depth information labels.

[0022] It should be noted that the semantic categories include various types of obstacles and non-obstacles. The various types of obstacles are divided into step-like obstacles and non-step-like obstacles according to the standard height of the obstacles. Among them, the standard height of the non-step-like obstacles is higher than that of the step-like obstacles.

[0023] For the convenience of description, take the semantic categories including a drivable area, a curb, special targets, and a background as an example. Among them, the drivable area corresponds to the area flush with the plane where the vehicle is located, that is, a non-obstacle. The curb corresponds to the area perpendicular to the ground, with the bottom flush with the drivable area and the top flush with the pedestrian area, that is, a step-like obstacle. The special targets correspond to targets close to the height of the car door. The special targets include, but are not limited to, a ground lock and a jack. The background corresponds to other types of targets. The background includes, but is not limited to, buildings, trees, and vehicles. Among them, in other specific implementation scenarios, the semantic categories can also be specifically set based on the actual scenario, and the present application does not make specific limitations on this.

[0024] S102: Based on the coordinates matched by multiple pixels in the image area, determine the detection height corresponding to the step-like obstacle. Based on the coordinates matched by the pixels at the specified position in the image area and the depth information included in the depth image, determine the detection height corresponding to the non-step-like obstacle.

[0025] Specifically, based on the coordinates matched by multiple pixels in the image area, analyze the height of the step-like obstacle to determine the detection height corresponding to the step-like obstacle, so that when the height difference of the step-like obstacle changes little and the depth information is not obvious, the detection height of the obstacle is determined by multi-point sampling.

[0026] In some implementation scenarios, obtain the installation height of the acquisition device that acquires the to-be-detected image, and the calibration coordinates of the acquisition device in the vehicle coordinate system. Obtain the pixel coordinates matched by the multiple pixels corresponding to the upper end from the image area corresponding to the step-like obstacle, convert the pixel coordinates to the vehicle coordinate system to obtain the converted coordinates. Based on the installation height, calibration coordinates, and converted coordinates, fit the height of the pixel in the vehicle coordinate system to determine the height matched by the pixel. Based on the heights matched by multiple pixels, determine the detection height corresponding to the step-like obstacle.

[0027] In some implementation scenarios, multiple groups of pixel pairs are obtained from the image area corresponding to the step-like obstacle. Among them, each pixel pair includes two pixels that are vertically located at the upper edge and the lower edge. Based on the image size and the pixel difference corresponding to the pixels in the pixel pair, the height differences corresponding to multiple groups of pixel pairs are obtained. Based on the multiple groups of height differences, the detection height corresponding to the step-like obstacle is determined.

[0028] It can be understood that based on the coordinates matched by the pixels at the specified position in the image area and the depth information included in the depth image, the height of the non-step-like obstacle is analyzed to determine the detection height corresponding to the non-step-like obstacle. Thus, when the height drop of the non-step-like obstacle changes greatly, the height of the non-step-like obstacle in the three-dimensional space is detected by combining the depth information, and finally the accuracy of the detection height corresponding to the obstacles with different height drops is improved.

[0029] In some implementation scenarios, the highest point in the image area corresponding to the non-step-like obstacle is taken as the specified position, the pixel coordinates matched by the pixels at the specified position are determined. Based on the depth information of the specified pixel and the device parameters of the acquisition device, the pixel coordinates of the specified pixel are converted into the vehicle coordinate system corresponding to the vehicle to obtain the target conversion coordinates corresponding to the specified pixel. Based on the target conversion coordinates, the detection height corresponding to the non-step-like obstacle is determined.

[0030] In some implementation scenarios, the highest point and the lowest point in the image area corresponding to the non-step-like obstacle are taken as the specified positions, the installation height of the acquisition device for acquiring the image to be detected is obtained. Based on the depth information, the plane where the specified position is located is determined. Based on the plane distance between the plane where the specified position is located and the plane where the acquisition device is located, and the installation height of the acquisition device, the height difference between the highest point and the lowest point is determined, and the corresponding height difference is used as the detection height corresponding to the non-step-like obstacle.

[0031] S103: Based on the detection height corresponding to each type of obstacle and the reference height corresponding to the vehicle door, determine the target opening parameter corresponding to the vehicle door; wherein, the target opening parameter is used to avoid obstacles higher than the reference height when the vehicle door is opened.

[0032] Specifically, based on the detection height corresponding to each type of obstacle and the reference height corresponding to the vehicle door, analyze how to set the vehicle door parameters to determine the target opening parameter corresponding to the vehicle door, so that when the vehicle door is opened according to the target opening parameter, it can avoid obstacles higher than the reference height, thereby reducing the probability of collision between the opened vehicle door and the obstacle and improving the accuracy of vehicle door obstacle avoidance.

[0033] In some implementation scenarios, based on the detection height corresponding to various obstacles and the reference height corresponding to the vehicle door, the area of the image to be detected in the vertical projection plane is divided into a collision area and a non - collision area. Among them, the detection height of the collision area exceeds the reference height, and the height of the non - collision area does not exceed the reference height. Based on the collision area, the non - collision area, and the area swept by the vehicle door on the projection plane when the door is opened, the target door opening parameters corresponding to the vehicle door are determined.

[0034] In some implementation scenarios, based on the detection height corresponding to various obstacles and the depth image, a three - dimensional model corresponding to the scene matched with the image to be detected is constructed, and the opening and closing area swept by the vehicle door from the closed state to the fully open state in the three - dimensional model is obtained. Based on the detection height of the obstacles in the opening and closing area and the reference height corresponding to the vehicle door, the target door opening parameters corresponding to the vehicle door are determined.

[0035] It can be understood that the reference height corresponding to the vehicle door is the height of the lowest point of the vehicle door from the plane where the vehicle is located, and the target door opening parameters at least include the final opening angle.

[0036] In the above solution, the image to be detected is obtained, the semantic segmentation image and the depth image corresponding to the image to be detected are determined. Different semantic categories corresponding to different image areas can be determined from the semantic segmentation image, and the depth information corresponding to the pixels can be obtained from the depth image. Multiple semantic categories correspond to different types of obstacles, and the multiple types of obstacles include step - type obstacles and non - step - type obstacles. Based on the coordinates matched by multiple pixels in the image area, the height of the step - type obstacles is analyzed to determine the detection height corresponding to the step - type obstacles. Thus, when the height difference of the step - type obstacles changes little and the depth information difference is not obvious, the detection height of the obstacles is determined by multi - point sampling. Based on the coordinates matched by the pixels at the specified position in the image area and the depth information included in the depth image, the height of the non - step - type obstacles is analyzed to determine the detection height corresponding to the non - step - type obstacles. Thus, when the height difference of the non - step - type obstacles changes greatly, the height of the non - step - type obstacles in the three - dimensional space is detected in combination with the depth information, and finally the accuracy of the detection height corresponding to the obstacles with different height differences is improved. Based on the detection height corresponding to various obstacles and the reference height corresponding to the vehicle door, how to set the door parameters is analyzed to determine the target door opening parameters corresponding to the vehicle door, so that when the door is opened according to the target door opening parameters, the obstacles higher than the reference height can be avoided, thereby reducing the probability of collision between the opened door and the obstacles and improving the accuracy of door obstacle avoidance.

[0037] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of another implementation manner of the vehicle door obstacle avoidance method of the present application. The method includes:

[0038] S201: Obtain the semantic segmentation image and depth image corresponding to the image to be detected; among them, the semantic segmentation image includes the semantic categories corresponding to the image regions, the depth image includes the depth information corresponding to the pixels, and the obstacles matching multiple semantic categories include step-like obstacles and non-step-like obstacles.

[0039] Specifically, obtain the image to be detected, determine the semantic segmentation image and depth image corresponding to the image to be detected. The semantic categories corresponding to different image regions can be determined from the semantic segmentation image, and the depth information corresponding to the pixels can be obtained from the depth image. Different categories of obstacles correspond to multiple semantic categories, and the multiple categories of obstacles include step-like obstacles and non-step-like obstacles.

[0040] S202: Based on the coordinates matched by multiple pixels in the image region, determine the detection height corresponding to the step-like obstacle, and based on the coordinates matched by the pixels at the specified position in the image region and the depth information included in the depth image, determine the detection height corresponding to the non-step-like obstacle.

[0041] Specifically, analyze the height of the step-like obstacle based on the coordinates matched by multiple pixels in the image region to determine the detection height corresponding to the step-like obstacle, and analyze the height of the non-step-like obstacle based on the coordinates matched by the pixels at the specified position in the image region and the depth information included in the depth image to determine the detection height corresponding to the non-step-like obstacle.

[0042] In some implementation scenarios, determining the detection height corresponding to the step-like obstacle based on the coordinates matched by multiple pixels in the image region includes: obtaining the boundary pixels at the upper and lower ends of the step-like obstacle in the image region, and determining the coordinates matched by the boundary pixels; obtaining the installation height of the acquisition device for acquiring the image to be detected, and based on the installation height and the coordinates matched by the boundary pixels, determining the detection height corresponding to the step-like obstacle.

[0043] Specifically, obtain the boundary pixels at the upper and lower ends of the step-like obstacle in the vertical section of the image region, determine the pixel coordinates of the boundary pixels in the image to be detected and convert them to the vehicle coordinate system to obtain the coordinates matched by the boundary pixels.

[0044] It should be noted that the acquisition device has device parameters, which include the internal parameter matrix and distortion coefficient of the acquisition device, as well as the external parameter matrix and translation matrix of the acquisition device. Based on the internal parameter matrix and distortion coefficient of the acquisition device, the pixel coordinates are converted to the normalized device coordinate system, and based on the external parameter matrix and translation matrix of the acquisition device, the plane coordinates in the normalized device coordinate system are converted to the vehicle coordinate system.

[0045] Further, determine the installation height corresponding to the acquisition device for acquiring the image to be detected. Based on the installation height and the coordinates of the boundary pixel matching, fit the height of the vertical plane of the step-like obstacle to obtain the detection height corresponding to the step-like obstacle, so as to detect the height of the step-like obstacle in combination with the installation height of the acquisition device when the height difference of the step-like obstacle changes little and the depth information is not obvious, and improve the accuracy of the detection height of the step-like obstacle.

[0046] In a specific implementation scenario, to obtain the installation height corresponding to the acquisition device for acquiring the image to be detected and determine the detection height corresponding to the step-like obstacle based on the installation height and the coordinates of the boundary pixel matching, the following steps are included: Based on the coordinates of the boundary pixel matching at the lower end, determine the plane where the step-like obstacle is located, determine the installation height of the acquisition device relative to the plane, and the projection coordinates of the acquisition device projected onto the plane; Based on the installation height and the coordinates of the boundary pixel matching at the upper end, determine the intersection point of the line connecting the acquisition device and the boundary pixel at the upper end with the plane, and obtain the coordinates of the intersection point matching; Based on the installation height, the projection coordinates, the coordinates of the boundary pixel matching at the lower end, and the coordinates of the intersection point, determine the detection height corresponding to the step-like obstacle.

[0047] Specifically, obtain the coordinates of the boundary pixel matching at the lower end, determine the plane where the step-like obstacle is located based on the corresponding coordinates, determine the installation height of the acquisition device relative to the plane where the obstacle is located to make the installation height more accurate, and determine the projection coordinates of the acquisition device projected onto the plane.

[0048] Further, based on the installation height of the acquisition device and the coordinates of the boundary pixel matching at the upper end, determine the line connecting the two between the acquisition device and the boundary pixel at the upper end, obtain the intersection point of the line with the plane where the obstacle is located and the coordinates of the intersection point matching. Then, based on the installation height, the projection coordinates, the coordinates of the boundary pixel matching at the lower end, and the coordinates of the intersection point, fit the height of the step-like obstacle to determine the detection height corresponding to the step-like obstacle and improve the accuracy of the detection height corresponding to the step-like obstacle.

[0049] For ease of explanation, Figure 3 is a schematic diagram of the application scenario of an implementation manner for obtaining the detection height of the step-like obstacle in this application. represents the installation height of the acquisition device. represents the detection height of the step-like obstacle. The projection point of the acquisition device on the ground is , and the projection point of the step-like obstacle on the ground is . The intersection point of the line connecting the acquisition device and the vertex of the step-like obstacle with the ground is . In the vehicle coordinate system, assume the coordinates of each point are: , , , the detection height can be obtained based on the following formula:

[0050] (1)

[0051] where D represents the distance between coordinates.

[0052] In some implementation scenarios, based on the coordinates of pixel matching at a specified position in the image area and the depth information included in the depth image, determining the detection height corresponding to non-step obstacles includes: obtaining the specified pixel at the specified position of the non-step obstacle in the image area, and determining the pixel coordinates of the specified pixel in the image to be detected; obtaining the device parameters of the acquisition device that acquires the image to be detected, and based on the depth information and device parameters of the specified pixel, converting the pixel coordinates of the specified pixel into the vehicle coordinate system corresponding to the vehicle to obtain the target conversion coordinates corresponding to the specified pixel; and determining the detection height corresponding to the non-step obstacle based on the target conversion coordinates.

[0053] Specifically, obtain the specified pixel at the specified position from the image area corresponding to the non-step obstacle, determine the pixel coordinates of the specified pixel in the image to be detected, obtain the device parameters matched by the acquisition device that acquires the image to be detected, and use the depth information and device parameters of the specified pixel in the depth image to convert the pixel coordinates of the specified pixel into the vehicle coordinate system corresponding to the vehicle. Thus, when the height difference of the non-step obstacle changes greatly, the specified pixel at the specified position is converted into the vehicle coordinate system in combination with the depth information to obtain the target conversion coordinates, so that the values of the upper and lower spaces in the target conversion coordinates can reflect the height of the specified pixel in the three-dimensional space.

[0054] It can be understood that based on the values of the upper and lower spaces in the target conversion coordinates, that is, based on the values of the Z-axis, the detection height corresponding to the non-step obstacle can be determined.

[0055] It should be noted that the specified position can be the position corresponding to the highest point in the image area, so as to uniformly identify the pixels in the corresponding image area as the height corresponding to the highest point to adapt to background targets such as walls or vehicles. Or, the specified position can be the position corresponding to some identification points in the target contour in the image area, so as to detect the shape of the target in the image area and the height of the target with the corresponding shape at different identification points to adapt to special targets such as ground locks or jacks.

[0056] In a specific implementation scenario, the device parameters include the internal parameter matrix and distortion coefficients of the acquisition device, as well as the external parameter matrix and translation matrix of the acquisition device; based on the depth information of a specified pixel and the device parameters, the pixel coordinates of the specified pixel are converted into the corresponding vehicle coordinate system of the vehicle to obtain the target conversion coordinates corresponding to the specified pixel, including: based on the internal parameter matrix and distortion coefficients of the acquisition device, the pixel coordinates are converted into the normalized device coordinate system to obtain the planar conversion coordinates and the incident angle; based on the planar conversion coordinates, the incident angle and the depth information of the specified pixel, the three-dimensional conversion coordinates of the specified pixel in the device coordinate system are determined; based on the three-dimensional conversion coordinates and the external parameter matrix and translation matrix of the acquisition device, the target conversion coordinates of the specified pixel in the vehicle coordinate system are determined.

[0057] Specifically, assuming that the pixel coordinates corresponding to the specified pixel are , according to the internal parameter matrix and distortion coefficients, the planar conversion coordinates of this point in the normalized device coordinate system can be obtained and the incident angle . Given that the depth value of the specified pixel included in the depth image is D. Then the three-dimensional conversion coordinates of this point in the device coordinate system are:

[0058] (2)

[0059] (3)

[0060] (4)

[0061] where sqrt represents taking the square root.

[0062] Furthermore, according to the external parameter matrix and the translation matrix , the target conversion coordinates in the vehicle coordinate system can be calculated, where . Based on the value in the target conversion coordinates, the detection height corresponding to non-step obstacles can be determined, so that by combining the depth information and the device parameters of the acquisition device, the accurate detection height of non-step obstacles can be obtained, and the target conversion coordinates in the vehicle coordinate system can reflect the relative position relationship between non-step obstacles and the vehicle door, thus facilitating the judgment of whether a collision will occur.

[0063] S203: Based on the detection heights corresponding to various obstacles and the reference height corresponding to the vehicle door, determine the collision area and non-collision area in the image to be detected; among them, the detection height of the collision area exceeds the reference height, and the height of the non-collision area does not exceed the reference height.

[0064] Specifically, obtain the reference height corresponding to the vehicle door. Based on the height differences between the detection heights of various obstacles and the reference height corresponding to the vehicle door, determine the collision area and non-collision area in the image to be detected, so as to obtain the collision area where the detection height exceeds the reference height and the non-collision area where the detection height does not exceed the reference height, facilitating the judgment of whether a collision will occur when opening the door.

[0065] In some implementation scenarios, based on the detection heights corresponding to various obstacles and the reference height corresponding to the vehicle door, determining the collision area and non-collision area in the image to be detected includes: based on the detection heights corresponding to various obstacles, setting matching height values for the pixels in the corresponding image areas to generate a height map corresponding to the image to be detected; based on the height values matched by the pixels in the height map and the reference height, determining the collision area and non-collision area in the image to be detected.

[0066] Specifically, based on the detection heights corresponding to various obstacles, set matching height values for the pixels in the corresponding image areas to generate a height map matching the image to be detected, where the distance between the obstacles in the vehicle coordinate system and the vehicle is corresponding in the height map, so as to generate the heights of different targets in the space within the space corresponding to the image to be detected.

[0067] Furthermore, traverse the pixels in the height map, compare the height values matched by the pixels in the height map with the reference height, take the area where the detection height exceeds the reference height as the collision area, and take the area where the detection height does not exceed the reference height as the non-collision area, improving the accuracy of area division.

[0068] S204: Obtain the reference opening area corresponding to the vehicle door. Based on the intersection of the reference opening area and the collision area, determine the target opening parameter corresponding to the vehicle door; where the reference opening area corresponds to the maximum opening parameter of the vehicle door.

[0069] Specifically, obtain the reference opening area corresponding to the vehicle door when it is opened according to the maximum opening parameter, obtain the intersection of the reference opening area and the collision area. When the intersection is an empty set, take the maximum opening parameter as the target opening parameter. When the intersection is a non-empty set, based on the intersection of the reference opening area and the collision area, re-determine the target opening parameter that the vehicle door will not collide with the collision area when opened, so as to determine the target opening parameter only based on the intersection area, improving the efficiency of obtaining the target opening parameter.

[0070] It can be understood that when the reference opening area is mapped on the projection plane, it corresponds to the sector area swept by the vehicle door from the closed state to the maximum opening state.

[0071] It should be noted that the obstacle to be confirmed corresponding to the obstacle in the reference opening area is considered; after obtaining the reference opening area corresponding to the vehicle door and determining the target opening parameters corresponding to the vehicle door based on the intersection of the reference opening area and the collision area, the following steps are further included: in response to the absolute difference between the detected height of the obstacle to be confirmed and the reference height being less than the difference threshold, generating parameter display information matching the semantic category and the detected height of the obstacle to be confirmed, and generating execution prompt information corresponding to the target opening parameters.

[0072] Specifically, the obstacles in the reference opening area need to be focused on. The obstacles in the reference opening area are marked as obstacles to be confirmed, and the absolute difference between the detected height of the obstacle to be confirmed and the reference height is determined. When the absolute difference is less than the difference threshold, parameter display information is generated using the semantic category and the detected height of the obstacle to be confirmed, so as to display the category of the obstacle to be confirmed and the detected height of the obstacle to be confirmed through the parameter display information. Among them, the parameter display information is usually displayed on the in-vehicle screen.

[0073] Furthermore, execution prompt information corresponding to the target opening parameters is generated, so as to prompt whether to execute the current target opening parameters through the execution prompt information, reducing the probability of damage to the vehicle door caused by forced opening due to inaccurate detected height.

[0074] In this embodiment, boundary pixels at the upper and lower ends of the vertical section of the step-like obstacle in the image area are obtained, the pixel coordinates of the boundary pixels in the image to be detected are determined and converted into the vehicle coordinate system to obtain the coordinates corresponding to the boundary pixels, the installation height corresponding to the acquisition device for acquiring the image to be detected is determined, and based on the installation height and the coordinates corresponding to the boundary pixels, the height of the vertical plane of the step-like obstacle is fitted to obtain the detected height corresponding to the step-like obstacle. When the height difference of the step-like obstacle changes little and the depth information is not obvious, the height of the step-like obstacle can be detected in combination with the installation height of the acquisition device, so as to improve the accuracy of the detected height of the step-like obstacle. Designated pixels at designated positions are obtained from the image area corresponding to non-step-like obstacles, the pixel coordinates of the designated pixels in the image to be detected are determined, the device parameters matched with the acquisition device for acquiring the image to be detected are obtained, and by using the depth information of the designated pixels in the depth image and the device parameters, the pixel coordinates of the designated pixels are converted into the vehicle coordinate system corresponding to the vehicle. Thus, when the height difference of the non-step-like obstacle changes greatly, the designated pixels at the designated positions are converted into the vehicle coordinate system in combination with the depth information to obtain the target conversion coordinates, so that the values in the upper and lower spaces in the target conversion coordinates can reflect the height of the designated pixels in the three-dimensional space. Based on the detected height corresponding to each type of obstacle and the reference height corresponding to the vehicle door, the collision area and non-collision area in the image to be detected are determined, the reference opening area corresponding to the vehicle door when it is opened according to the maximum opening parameter is obtained, and the intersection of the reference opening area and the collision area is obtained. The target opening parameter can be determined only based on the intersection area, which improves the efficiency of obtaining the target opening parameter.

[0075] Please refer to Figure 4 , Figure 4 FIG. is a schematic structural diagram of an embodiment of an electronic device according to the present application. The electronic device 30 includes a memory 301 and a processor 302 which are coupled to each other. Among them, the memory 301 stores program data (not shown in the figure), and the processor 302 calls the program data to implement the method in any of the above embodiments. For the description of related content, please refer to the detailed description of the above method embodiments and will not be repeated here.

[0076] Please refer to Figure 5 , Figure 5 FIG. is a schematic structural diagram of an embodiment of a computer-readable storage medium according to the present application. The computer-readable storage medium 40 stores program data 400, and when the program data 400 is executed by a processor, it implements the method in any of the above embodiments. For the description of related content, please refer to the detailed description of the above method embodiments and will not be repeated here.

[0077] It should be noted that the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0078] In addition, the functional units in various embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0079] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods in various embodiments of the present application. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs and other various media that can store program codes.

[0080] The above are only the embodiments of the present application, and do not limit the protection scope of the present application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be equally included in the protection scope of the present application.

Claims

1. A vehicle door obstacle avoidance method, characterized in that: The method comprises: Obtain a semantic segmentation image and a depth image corresponding to the image to be detected; wherein the semantic segmentation image includes the semantic category corresponding to the image area, the depth image includes the depth information corresponding to the pixel, and obstacles matching multiple semantic categories include step obstacles and non-step obstacles; Determine the detection height corresponding to the step obstacle based on the coordinates of the multiple pixel matches in the image area, and determine the detection height corresponding to the non-step obstacle based on the coordinates of the pixel matches at a specified position in the image area and the depth information included in the depth image; Based on the detection heights corresponding to various obstacles and the reference heights corresponding to the vehicle doors, target door opening parameters corresponding to the vehicle doors are determined; wherein the target door opening parameters are used to avoid obstacles higher than the reference height when the vehicle doors are opened.

2. The vehicle door obstacle avoidance method according to claim 1, characterized in that: The determining the detection height corresponding to the step obstacle based on the coordinates of the plurality of pixel matches in the image area includes: Acquire boundary pixels of the step-like obstacle at the upper and lower ends in the image region, and determine the coordinates of the boundary pixel matches; The installation height corresponding to the acquisition device that acquires the image to be detected is obtained, and the detection height corresponding to the step-type obstacle is determined based on the installation height and the coordinates of the boundary pixel matching.

3. The vehicle door obstacle avoidance method according to claim 2, characterized in that: The obtaining of the installation height corresponding to the acquisition device for acquiring the image to be detected, and determining the detection height corresponding to the step obstacle based on the installation height and the coordinates matched by the boundary pixels, includes: Based on the coordinates of the boundary pixel matching at the lower end, determine the plane where the step-like obstacle is located, determine the installation height of the acquisition device relative to the plane, and determine the projection coordinates of the acquisition device projected onto the plane; Based on the coordinates of the installation height and the boundary pixel at the upper end, determine the intersection of the line between the acquisition device and the boundary pixel at the upper end and the plane, and obtain the coordinates of the intersection; The detection height corresponding to the step-type obstacle is determined based on the installation height, the projection coordinates, the coordinates of the boundary pixel matching at the lower end, and the coordinates of the intersection.

4. The vehicle door obstacle avoidance method according to claim 1, characterized in that: The determining the detection height corresponding to the non-step obstacle based on the coordinates of the pixel matching at the designated position in the image area and the depth information included in the depth image includes: Acquire a designated pixel of the non-step obstacle in the image region at a designated position, and determine the pixel coordinates of the designated pixel in the image to be detected; Acquire device parameters of a collection device that collects the image to be detected, and based on the depth information and device parameters of the designated pixel, convert the pixel coordinates of the designated pixel into a vehicle coordinate system corresponding to the vehicle to obtain a target conversion coordinate corresponding to the designated pixel; Based on the target conversion coordinates, a detection height corresponding to the non-step obstacle is determined.

5. The vehicle door obstacle avoidance method according to claim 4, characterized in that: The device parameters include the intrinsic parameter matrix and distortion coefficient of the acquisition device, and the extrinsic parameter matrix and translation matrix of the acquisition device; The step of converting the pixel coordinates of the designated pixel into a vehicle coordinate system corresponding to the vehicle based on the depth information and device parameters of the designated pixel to obtain a target conversion coordinate corresponding to the designated pixel includes: Based on the intrinsic parameter matrix and distortion coefficient of the acquisition device, the pixel coordinates are converted into a normalized device coordinate system to obtain a plane conversion coordinate and an incident angle; Determine a three-dimensional transformation coordinate of the designated pixel transformed into a device coordinate system based on the plane transformation coordinate, the incident angle and the depth information of the designated pixel; Based on the three-dimensional transformation coordinates and the external parameter matrix and the translation matrix of the acquisition device, the target transformation coordinates of the designated pixel in the vehicle coordinate system are determined.

6. The vehicle door obstacle avoidance method according to claim 1, characterized in that: The determining of the target door opening parameters corresponding to the door based on the detection heights corresponding to the various obstacles and the reference heights corresponding to the door includes: Based on the detection heights corresponding to various obstacles and the reference height corresponding to the vehicle door, determining the collision area and the non-collision area in the image to be detected; wherein the detection height of the collision area exceeds the reference height, and the height of the non-collision area does not exceed the reference height; A reference door opening area corresponding to the vehicle door is obtained, and a target door opening parameter corresponding to the vehicle door is determined based on an intersection of the reference door opening area and the collision area; wherein the reference door opening area corresponds to a maximum door opening parameter corresponding to the vehicle door.

7. The vehicle door obstacle avoidance method according to claim 6, characterized in that: The determining of the collision area and the non-collision area in the image to be detected based on the detection heights corresponding to the various obstacles and the reference height corresponding to the vehicle door comprises: Based on the detection heights corresponding to various types of obstacles, matching height values ​​are set for pixels in the corresponding image area to generate a height map corresponding to the image to be detected; Based on the height values ​​of the pixel matches in the height map and the reference height, a collision area and a non-collision area in the image to be detected are determined.

8. The vehicle door obstacle avoidance method according to claim 6, characterized in that: Obstacles to be confirmed corresponding to the obstacles in the reference door opening area; After obtaining the reference door opening area corresponding to the vehicle door and determining the target door opening parameter corresponding to the vehicle door based on the intersection of the reference door opening area and the collision area, the method further includes: In response to the absolute difference between the detected height of the obstacle to be confirmed and the reference height being less than a difference threshold, parameter display information matching the semantic category and detected height corresponding to the obstacle to be confirmed is generated, and execution prompt information corresponding to the target door opening parameter is generated.

9. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images