A method, apparatus, device, and storage medium for rendering obstacles.

Abstract

This invention discloses an obstacle rendering method, apparatus, device, and storage medium, relating to the field of device control technology. The method includes: acquiring image data and point cloud data collected by an unmanned aerial vehicle (UAV); identifying semantic information in the image data and determining a first position of the obstacle in the image data based on the semantic information; determining a second position of the obstacle relative to the UAV based on the point cloud data; and rendering the obstacle in the image data according to preset rendering rules, based on the first and second position information, to obtain a rendered image. This technical solution, by rendering obstacles in image data according to preset rendering rules, can meet the obstacle avoidance and early warning needs of UAVs with limited computing power, rationally allocate obstacle rendering resources, thereby assisting users in safe flight control, and is applicable to various UAVs.

Description

Technical Field

[0001] This application relates to the field of equipment control technology, and in particular to a method, apparatus, device and medium for rendering obstacles. Background Technology

[0002] Unmanned aerial vehicles (UAVs) transmit real-time footage captured by FPV (First Person View) cameras to a user control terminal, allowing users to understand the surrounding environment by viewing the live feed. During its movement, the UAV can perceive depth information to help determine obstacles along its path and provide guidance to the user.

[0003] To provide users with a more intuitive understanding of obstacle information, existing technologies render obstacle information and overlay the rendering results directly onto the real-time screen for direct viewing. However, as the market for unmanned aerial vehicles (UAVs) gradually expands to lower-tier cities, the computing resources and performance offered by UAVs are gradually decreasing to meet cost requirements. This results in UAVs being unable to render all obstacle information.

[0004] Therefore, how to reasonably allocate rendering resources for obstacles and meet the computing power limitations of unmanned aerial vehicles is a problem that urgently needs to be solved by people in this field. Summary of the Invention

[0005] This invention provides an obstacle rendering method, apparatus, device, and medium. When the computing power of unmanned aerial vehicles (UAVs) is limited, it meets the obstacle avoidance and early warning needs of UAVs, rationally allocates obstacle rendering resources, thereby assisting users in safe flight control, and is applicable to various UAVs.

[0006] In a first aspect, embodiments of the present invention provide a method for rendering obstacles, the method being executed by an unmanned aerial vehicle, the method comprising:

[0007] Acquire image data and point cloud data collected by unmanned aerial vehicles;

[0008] Identify the semantic information of the image data, and determine the first position information of the obstacle in the image data based on the semantic information;

[0009] The second position information of the obstacle relative to the unmanned aerial vehicle is determined based on the point cloud data;

[0010] According to preset rendering rules, the obstacle is rendered in the image data based on the first position information and the second position information to obtain a rendered image.

[0011] Secondly, embodiments of the present invention also provide an obstacle rendering apparatus, the apparatus comprising:

[0012] The data acquisition module is used to acquire image data and point cloud data collected by the unmanned aerial vehicle.

[0013] The first position determination module is used to identify the semantic information of the image data and determine the first position information of the obstacle in the image data based on the semantic information.

[0014] The second position determination module is used to determine the second position information of the obstacle relative to the unmanned aerial vehicle based on the point cloud data;

[0015] The image rendering module is used to render the obstacle in the image data based on the first position information and the second position information according to the preset rendering rules, so as to obtain a rendered image.

[0016] Thirdly, embodiments of the present invention also provide an unmanned aerial vehicle (UAV) device, the UAV device comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the obstacle rendering method as described in the first aspect.

[0017] Fourthly, embodiments of the present invention also provide a storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform the steps of the obstacle rendering method as described in the first aspect.

[0018] In this embodiment of the invention, image data and point cloud data collected by an unmanned aerial vehicle (UAV) are acquired; semantic information of the image data is identified, and a first position information of an obstacle in the image data is determined based on the semantic information; a second position information of the obstacle relative to the UAV is determined based on the point cloud data; and the obstacle is rendered in the image data based on the first and second position information according to a preset rendering rule to obtain a rendered image. This obstacle rendering method, by rendering obstacles in the image data according to preset rendering rules, can meet the obstacle avoidance and early warning needs of UAVs with limited computing power, rationally allocate obstacle rendering resources, thereby assisting users in safe flight control, and is applicable to various UAVs. Attached Figure Description

[0019] Figure 1 A flowchart illustrating the obstacle rendering method provided in Embodiment 1 of this application;

[0020] Figure 2 This is a horizontal plane schematic diagram of the field of view of the image data of the unmanned aerial vehicle provided in Embodiment 1 of this application;

[0021] Figure 3 This is a flowchart illustrating the obstacle rendering method provided in Embodiment 2 of the present invention;

[0022] Figure 4 This is a horizontal plane schematic diagram of the preset rendering area of ​​the unmanned aerial vehicle provided in Embodiment 2 of this application;

[0023] Figure 5 This is a horizontal plan view of the preset warning area of ​​the unmanned aerial vehicle provided in Embodiment 2 of this application;

[0024] Figure 6 This is a flowchart illustrating the obstacle rendering method provided in Embodiment 3 of the present invention;

[0025] Figure 7 This is a schematic diagram of a horizontal plane divided according to the distance in front of the flight, provided in Embodiment 3 of this application;

[0026] Figure 8 This is a schematic diagram of a vertical plane divided according to the distance above and below the flight path, provided in Embodiment 3 of this application;

[0027] Figure 9 This is a flowchart illustrating the obstacle rendering method provided in Embodiment 4 of the present invention;

[0028] Figure 10 This is an example image of a rendered image provided in Embodiment 4 of the present invention;

[0029] Figure 11 This is a schematic diagram of the obstacle rendering device provided in Embodiment 5 of this application;

[0030] Figure 12 This is a structural schematic diagram of an unmanned aerial vehicle provided in Embodiment Six of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0032] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0033] The obstacle rendering method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0034] Example 1

[0035] Figure 1 This is a flowchart illustrating the obstacle rendering method provided in Embodiment 1 of this application. Figure 1 As shown, the specific steps include the following:

[0036] S101. Acquire image data and point cloud data collected by the unmanned aerial vehicle;

[0037] The application scenario for this solution can be a scenario where unmanned aerial vehicles identify obstacles and provide prompts to users during movement.

[0038] Based on the above usage scenarios, it is understood that the solution in this application can be executed by the unmanned aerial vehicle, by the user control terminal, or by a combination of both. No further limitations are made here.

[0039] Unmanned aerial vehicles (UAVs) can refer to devices that can move in the air without a pilot and can perform tasks or commands issued by users, such as agricultural drones.

[0040] Image data refers to data collected by the FPV camera on an unmanned aerial vehicle (UAV). Sending this image data to the user control terminal allows users to understand the environmental conditions of the UAV in real time, providing intuitive information for decision-making and operation. An FPV camera is a camera specifically designed for UAVs or other remotely controlled models; it typically has a wide-angle lens, providing a broad field of view.

[0041] Figure 2 This is a horizontal plane schematic diagram of the field of view of the image data of the unmanned aerial vehicle provided in Embodiment 1 of this application. For example... Figure 2As shown, the fan-shaped area represents the field of view of the image data of the unmanned aerial vehicle on the horizontal plane.

[0042] Point cloud data can refer to the relative position information of an object with respect to an unmanned aerial vehicle (UAV), that is, its position coordinates in a point cloud coordinate system. For example, the x-axis of the point cloud coordinate system points in front of the UAV, the y-axis points to the right of the UAV, and the z-axis points below the UAV.

[0043] Point cloud data can be acquired using millimeter-wave radar mounted on unmanned aerial vehicles (UAVs). Millimeter-wave radar is a sensor that uses electromagnetic waves in the millimeter-wave frequency band for target detection and measurement. Alternatively, point cloud data can also be acquired using binocular cameras or lidar mounted on UAVs.

[0044] It is understandable that unmanned aerial vehicles collect image data and point cloud data frame by frame during flight, and the image data and point cloud data mentioned in this step are collected in the same frame.

[0045] S102. Identify the semantic information of the image data, and determine the first position information of the obstacle in the image data based on the semantic information;

[0046] The semantic information of image data can refer to the outlines and category information of objects in the image data.

[0047] Image data is input into a pre-trained artificial intelligence (AI) model. The AI ​​model outputs the masked regions of each object in the image data and their corresponding category information, thereby achieving semantic information recognition of the image data. The AI ​​model can be a semantic segmentation model, a widely used technique in computer vision that assigns a specific category label to each pixel in an image, thus achieving pixel-by-pixel classification.

[0048] The first location information can refer to the pixel coordinates of the obstacle in the image data.

[0049] Determining the initial location information of obstacles in image data based on semantic information can be achieved through semantic segmentation of the image data using an artificial intelligence model. This allows for the acquisition of the outlines and categories of all objects in the image data. The object's outline contains its initial location information within the image data. In one example, all identified objects can be considered obstacles, thus semantic segmentation provides the initial location information of obstacles in the image data. In another example, a global point cloud map can be projected onto the image data. If a point cloud corresponding to the outline of an object in the image data is projected onto it, that object is identified as an obstacle. Specifically, this can be achieved by acquiring flight control attitude data from an unmanned aerial vehicle (UAV), building a global point cloud map based on the flight control attitude data and point cloud data, and projecting the global point cloud map onto the image data. This determines the point clouds that can be projected onto the image data and their corresponding pixel coordinates. Objects within the masked area of ​​these projected point clouds are identified as obstacles, and the pixel coordinates of these point clouds are determined as the initial location information of the obstacle in the image data. The category information corresponding to the masked area is also assigned to the point cloud.

[0050] The global point cloud map refers to a three-dimensional spatial model projected onto a world coordinate system from each frame of point cloud data. As an example, the world coordinate system could be a northeast-northeast coordinate system with the origin of the unmanned aerial vehicle's starting point, the x-axis pointing north, the y-axis pointing east, and the z-axis pointing to the ground. Understandably, during the creation of the global point cloud map—that is, with the projection of each frame of point cloud data—the point clouds can be filtered and fused based on their temporal and positional relationships within the global point cloud map.

[0051] S103. Determine the second position information of the obstacle relative to the unmanned aerial vehicle based on the point cloud data;

[0052] The second position information of the obstacle relative to the unmanned aerial vehicle is a description of the spatial position of the obstacle relative to the unmanned aerial vehicle, which can be the position coordinates of the obstacle in the point cloud coordinate system.

[0053] One method for determining the second position information of an obstacle relative to an unmanned aerial vehicle (UAV) is to use point cloud data associated with the obstacle to determine the second position information of the obstacle relative to the UAV.

[0054] S104. According to the preset rendering rules, the obstacle is rendered in the image data based on the first position information and the second position information to obtain a rendered image.

[0055] The preset rendering rules specify how to render the obstacle in the image data based on the first and second position information of the obstacle.

[0056] Rendered images can refer to image data that has been overlaid with rendered obstacle information.

[0057] According to preset rendering rules, obstacles are rendered in image data based on first and second position information to obtain a rendered image. This can be achieved by determining the target obstacle located within the preset rendering area based on the second position information, and then rendering the target obstacle based on the first position information of the target obstacle in the image data to obtain a rendered image.

[0058] Optionally, in this technical solution, after obtaining the rendered image, the method further includes:

[0059] The rendered image is sent to the user control terminal for display on the user control terminal.

[0060] A user control terminal can refer to a terminal device operated by a user to issue control commands to unmanned aerial vehicles. Examples include remote controls, mobile phones, and tablets.

[0061] The rendered image can be sent to the user control terminal via a communication link.

[0062] The advantage of this setup is that the unmanned aerial vehicle (UAV) performs image rendering, and the rendered image is transmitted to the user control terminal. The user control terminal then only needs to display this rendered image, significantly reducing the computational demands on it. Furthermore, by sending the rendered image to the user control terminal for display, users can gain a more intuitive understanding of the UAV's environment and the specific details of any obstacles.

[0063] In this embodiment of the invention, image data and point cloud data collected by an unmanned aerial vehicle (UAV) are acquired; semantic information of the image data is identified, and a first position information of an obstacle in the image data is determined based on the semantic information; a second position information of the obstacle relative to the UAV is determined based on the point cloud data; and the obstacle is rendered in the image data based on the first and second position information according to a preset rendering rule to obtain a rendered image. This obstacle rendering method, by rendering obstacles in the image data according to preset rendering rules, can meet the obstacle avoidance and early warning needs of UAVs with limited computing power, rationally allocate obstacle rendering resources, thereby assisting users in safe flight control, and is applicable to various UAVs.

[0064] Example 2

[0065] Figure 3 This is a flowchart illustrating the obstacle rendering method provided in Embodiment 2 of this application. This solution makes a further improvement to the above embodiment, specifically: rendering the obstacle in the image data based on the first position information and the second position information according to a preset rendering rule to obtain a rendered image, including: determining a target obstacle located within a preset rendering area based on the second position information; and rendering the target obstacle based on the first position information of the target obstacle in the image data to obtain a rendered image.

[0066] like Figure 3 As shown, it specifically includes the following:

[0067] S301. Acquire image data and point cloud data collected by the unmanned aerial vehicle;

[0068] S302. Identify the semantic information of the image data, and determine the first position information of the obstacle in the image data based on the semantic information;

[0069] S303. Determine the second position information of the obstacle relative to the unmanned aerial vehicle based on the point cloud data;

[0070] S304. Determine the target obstacle located within the preset rendering area based on the second location information;

[0071] The preset rendering area can be a pre-defined area where obstacles within it need to be rendered. For example, it can be a rectangular area with a preset rendering length and a preset rendering width. The preset rendering length could be 30 meters, and the preset rendering width could be 10 meters.

[0072] Figure 4 This is a horizontal plane schematic diagram of the preset rendering area of ​​the unmanned aerial vehicle provided in Embodiment 2 of this application. For example... Figure 4 As shown, the preset rendering area is a rectangular area with a preset rendering length and a preset rendering width. The area within the preset rendering area and the field of view of the image data is also defined as... Figure 4 The dark gray area in the text.

[0073] Target obstacles can refer to specific obstacles that need to be rendered.

[0074] The method of determining the target obstacle within the preset rendering area based on the second position information can be as follows: determine the vertical distance between the obstacle and the unmanned aerial vehicle (UAV) based on the second position information, which is the x-value of the obstacle in the point cloud coordinate system, and the horizontal distance between the obstacle and the UAV based on the second position information, which is the y-value of the obstacle in the point cloud coordinate system. If the x-value of the obstacle is within the preset rendering length and the y-value is within the preset rendering width, then the obstacle is determined as the target obstacle.

[0075] S305. Based on the first position information of the target obstacle in the image data, render the target obstacle to obtain a rendered image.

[0076] The method for rendering the target obstacle based on its first position information in the image data can be either to use a uniform rendering style to render the target obstacle at the first position information in the image data, or to determine the rendering style of the target obstacle based on its second position information relative to the unmanned aerial vehicle and then use the determined rendering style to render the target obstacle at the first position information in the image data to obtain a rendered image.

[0077] The advantage of this scheme is that by determining the target obstacle within the preset rendering area based on the second position information, and rendering the target obstacle based on the first position information of the target obstacle in the image data, the rendered image can be obtained. This can eliminate obstacles in the image data that do not affect the forward movement of the unmanned aerial vehicle, thus avoiding the waste of rendering resources caused by rendering these unaffected obstacles.

[0078] In this technical solution, optionally, the obstacle is rendered in the image data based on the first position information and the second position information according to a preset rendering rule to obtain a rendered image, and the solution further includes:

[0079] Based on the second location information, identify whether there are obstacles within the preset warning area;

[0080] If an obstacle is identified as being within a preset warning area, a warning message is rendered, resulting in a rendered image.

[0081] The preset warning zone can be a pre-defined area where a warning will be issued if an obstacle is present. Specifically, it can be a spherical area with a preset warning radius. The preset warning radius can be 5 meters.

[0082] Figure 5 This is a horizontal plane diagram of the preset warning area of ​​the unmanned aerial vehicle provided in Embodiment 2 of this application. For example... Figure 5As shown, the preset warning area is a spherical area with a preset warning radius as its radius, that is... Figure 5 The dark gray area is indicated by the dashed line.

[0083] The method of identifying whether there are obstacles in the preset warning area based on the second location information can be achieved by determining the straight-line distance between each obstacle and the unmanned aerial vehicle based on the second location information. If there are obstacles whose straight-line distance is less than the preset warning radius, they are identified as obstacles in the preset warning area.

[0084] Warning messages can be used to alert users that there are obstacles near the unmanned aerial vehicle (UAV) and that caution is required. One method for rendering warning messages when an obstacle is identified within a preset warning area is to render a red warning border at the corresponding position on the rendered image based on the obstacle's secondary position relative to the UAV.

[0085] The advantage of this solution is that by identifying whether there are obstacles within the preset warning area based on the second location information, and rendering warning information when obstacles are identified within the preset warning area, it can promptly and effectively issue danger warnings to users, enabling them to be aware of the potential risks that the unmanned aerial vehicle will face at the first moment.

[0086] Example 3

[0087] Figure 6 This is a flowchart illustrating the obstacle rendering method provided in Embodiment 3 of the present invention. This solution makes a further improvement to the above embodiment, specifically: rendering the target obstacle based on its first position information in the image data to obtain a rendered image, including: acquiring each directional interval pre-divided from the preset rendering region, and determining a preset rendering weight for each directional interval; rendering the target obstacle according to its first position information in the image data and the preset rendering weight of the directional interval where the target obstacle is located, to obtain a rendered image.

[0088] like Figure 6 As shown, it specifically includes the following:

[0089] S601. Acquire image data and point cloud data collected by the unmanned aerial vehicle;

[0090] S602. Identify the semantic information of the image data, and determine the first position information of the obstacle in the image data based on the semantic information;

[0091] S603. Determine the second position information of the obstacle relative to the unmanned aerial vehicle based on the point cloud data;

[0092] S604. Determine the target obstacle located within the preset rendering area based on the second location information;

[0093] S605. Obtain the directional intervals obtained by pre-dividing the preset rendering area, and determine the preset rendering weight of each directional interval.

[0094] The directional range can be the area obtained by dividing the preset rendering area.

[0095] Preset rendering weights can be pre-defined coefficients used to describe the rendering importance of various directional ranges.

[0096] The pre-defined rendering regions and their pre-defined rendering weights can be stored in the unmanned aerial vehicle (UAV). The pre-defined rendering weights of each region can then be obtained by reading the stored data from the UAV.

[0097] Optionally, in this technical solution, obtaining the pre-divided directional intervals of the preset rendering region and determining the preset rendering weights of each directional interval includes:

[0098] Obtain the directional intervals obtained by pre-dividing the preset rendering area, wherein the division method includes at least one of the following: division according to the distance in front of the flight, division according to the distance above and below the flight, and division according to the distance to the left and right of the flight.

[0099] Obtain the weight components of each directional interval under each division method;

[0100] Based on the weight components of the current directional interval under each division method, determine the preset rendering weight of the current directional interval, and traverse each directional interval to obtain the preset rendering weight of each directional interval.

[0101] The forward distance is the vertical distance between the drone and the unmanned aerial vehicle (UAV), which is the x-value in the point cloud coordinate system; the vertical distance is the height difference between the drone and the UAV, which is the z-value in the point cloud coordinate system; and the horizontal distance is the horizontal distance between the drone and the UAV, which is the y-value in the point cloud coordinate system.

[0102] The weight components of each directional interval under each partitioning method can be coefficients describing the rendering importance of each directional interval under each partitioning method. These weight components can be stored in association with the directional intervals themselves, meaning they can be retrieved through a query.

[0103] Figure 7This is a schematic diagram of a horizontal plane divided according to the distance in front of the flight path, provided in Embodiment 3 of this application. For example... Figure 7 As shown, the preset rendering area is divided according to the distance in front of the flight, and the weight components of each directional interval under this division method are as follows: the directional interval is the area with a distance in front of the flight between 0 and 5 meters, and its weight component is 5; the directional interval is the area with a distance in front of the flight between 5 and 10 meters, and its weight component is 3; the directional interval is the area with a distance in front of the flight between 10 and 20 meters, and its weight component is 1; the directional interval is the area with a distance in front of the flight between 20 and 30 meters, and its weight component is 1.

[0104] Figure 8 This is a schematic diagram of a vertical plane divided according to the vertical distance of flight, provided in Embodiment 3 of this application. For example... Figure 8 As shown, the preset rendering area is divided according to the vertical distance of flight, and the weight components of each directional interval under this division method are: the directional interval is the area with a vertical distance of flight between 0 and 1 meter, and its weight component is 7; the directional interval is the area with a vertical distance of flight between 1 and 2 meters, and its weight component is 2; the directional interval is the area with a vertical distance of flight between 2 and 3 meters, and its weight component is 1.

[0105] The preset rendering area is divided according to the left and right distances of flight, and the weight components of each directional interval under this division method are: the directional interval is the area between 0 and 1 meter of left and right distances of flight, and its weight component is 5; the directional interval is the area between 1 and 3 meters of left and right distances of flight, and its weight component is 3; the directional interval is the area between 3 and 5 meters of up and down distances of flight, and its weight component is 2.

[0106] The preset rendering weight of the current azimuth interval is determined based on the weight components of the current azimuth interval under each division method. This can be achieved by using the weight components of the current azimuth interval under one division method as the calculation base, determining the proportion of the weight components of the current azimuth interval under other division methods, and multiplying the calculation base by each proportion to obtain the preset rendering weight of the current azimuth interval.

[0107] For example, if a directional range is simultaneously located between 0 and 5 meters in front of the plane, between 1 and 2 meters above and below the plane, and between 0 and 1 meter to the left and right of the plane, then the formula for calculating the preset rendering weight of this directional range is:

[0108]

[0109] The advantage of this scheme is that by obtaining the pre-divided rendering area according to at least one division method, the weight components of each directional interval under each division method, and determining the pre-rendering weight of each directional interval according to the weight components of each directional interval under each division method, the rendering importance of obstacles at different distances in multiple directions can be fully considered, thereby improving the comprehensiveness and accuracy of the pre-rendering weight.

[0110] S606. Based on the first position information of the target obstacle in the image data, the target obstacle is rendered according to the preset rendering weight of the azimuth interval where the target obstacle is located, and a rendered image is obtained.

[0111] The method of rendering target obstacles according to the preset rendering weight of the target obstacle's location interval can be achieved by obtaining the number of target elements displayed by the unmanned aerial vehicle, calculating the sum of the preset rendering weights of the target obstacle's location interval, dividing the number of target elements displayed by the sum to obtain the number of elements displayed per unit rendering weight, multiplying the preset rendering weight of the target obstacle's location interval by the number of elements displayed per unit rendering weight for each target obstacle to obtain the number of target elements displayed for the target obstacle, and rendering the target obstacle according to the number of target elements displayed for each target obstacle based on the first position information of the target obstacle in the image data to obtain the rendered image.

[0112] In this technical solution, optionally, based on the first position information of the target obstacle in the image data, and according to a preset rendering weight of the azimuth interval where the target obstacle is located, the target obstacle is rendered to obtain a rendered image, including:

[0113] Obtain the number of elements displayed by the unmanned aerial vehicle;

[0114] Calculate the sum of the preset rendering weights for the location range of each target obstacle;

[0115] Divide the target number of elements to be displayed by the total to obtain the number of elements to be displayed per unit rendering weight;

[0116] For each target obstacle, the preset rendering weight of the directional interval where the target obstacle is located is multiplied by the number of elements displayed corresponding to the unit rendering weight to obtain the number of target elements displayed for the target obstacle.

[0117] For each target obstacle, based on the first position information of the target obstacle in the image data, the target obstacle is rendered according to the number of target elements displayed to obtain a rendered image.

[0118] The number of displayable elements refers to the pre-defined number of elements that the unmanned aerial vehicle (UAV) can render and display in a single frame. This number is understandably related to the UAV's computing power. An element can refer to a rendering unit, which is related to the rendering style. For example, if the rendering style is a 3D mesh, then one 3D mesh is considered one element.

[0119] The number of element display targets can be stored in the unmanned aerial vehicle (UAV), and then the number of element display targets can be obtained by reading the stored data of the UAV.

[0120] For example, assuming the number of elements to be displayed is 100, there is an obstacle A located simultaneously within 0 to 5 meters in front of the target, 1 to 2 meters above and below, and 0 to 1 meter to the left and right (preset rendering weight is 0.5); there is an obstacle B located simultaneously within 0 to 5 meters in front of the target, 1 to 2 meters above and below, and 0 to 1 meter to the left and right (preset rendering weight is 0.5); and there is an obstacle C located simultaneously within 5 to 10 meters in front of the target, 2 to 3 meters above and below, and 1 to 3 meters to the left and right (preset rendering weight is 0.09), then:

[0121] The sum of the preset rendering weights for the location range of each target obstacle = 0.5 + 0.5 + 0.09;

[0122]

[0123] The advantage of this scheme is that by calculating the sum of the preset rendering weights of the directional intervals of each target obstacle, dividing the number of target elements displayed by the sum, we obtain the number of elements displayed per unit rendering weight. Multiplying the preset rendering weight of the directional interval of the target obstacle by the number of elements displayed per unit rendering weight, we obtain the number of target elements displayed for the target obstacle. This fully considers the differences in rendering importance of target obstacles in different directional intervals and dynamically adjusts the number of elements displayed for each target obstacle according to the preset rendering weight, so that even with low computing power, we can highlight key obstacle information.

[0124] The advantage of this scheme is that by obtaining the pre-divided rendering area into various directional intervals, determining the pre-defined rendering weights of each directional interval, and rendering the target obstacle according to the first position information of the target obstacle in the image data and the pre-defined rendering weights of the directional interval where the target obstacle is located, the rendered image can be obtained. This allows for the reasonable allocation of rendering resources and highlights the rendering effect of obstacles in key directional intervals.

[0125] Example 4

[0126] Figure 9 This is a flowchart illustrating the obstacle rendering method provided in Embodiment 4 of the present invention. This solution makes further improvements to the above embodiments, specifically: after determining the target obstacle located within the preset rendering area based on the second position information, the method further includes: determining the rendering style of the target obstacle based on the second position information of the target obstacle relative to the unmanned aerial vehicle.

[0127] like Figure 9 As shown, it specifically includes the following:

[0128] S901: Acquire image data and point cloud data collected by unmanned aerial vehicles;

[0129] S902. Identify the semantic information of the image data, and determine the first position information of the obstacle in the image data based on the semantic information;

[0130] S903. Determine the second position information of the obstacle relative to the unmanned aerial vehicle based on the point cloud data;

[0131] S904. Determine the target obstacle located within the preset rendering area based on the second location information;

[0132] S905. Determine the rendering style of the target obstacle based on the second position information of the target obstacle relative to the unmanned aerial vehicle;

[0133] Rendering style can refer to a specific visual representation used to present a target obstacle.

[0134] The method of determining the rendering style of the target obstacle based on the second position information of the target obstacle relative to the unmanned aerial vehicle can be as follows: if the target obstacle is within the field of view of the image data, then the target obstacle's prompt shape information and / or prompt color information in the rendered image is determined based on the second position information of the target obstacle relative to the unmanned aerial vehicle and a preset distance level; or if the target obstacle is outside the field of view of the image data, then the target obstacle's proximity prompt shape information, prompt color information and / or prompt distance information in the rendered image is determined based on the second position information of the target obstacle relative to the unmanned aerial vehicle and a preset distance level.

[0135] In this technical solution, optionally, the rendering style of the target obstacle is determined based on the second position information of the target obstacle relative to the unmanned aerial vehicle, including:

[0136] If the target obstacle is within the field of view of the image data, then based on the second position information of the target obstacle relative to the unmanned aerial vehicle and a preset distance level, the prompt shape information and / or prompt color information of the target obstacle in the rendered image are determined.

[0137] Preset distance levels can refer to pre-defined standard levels used to distinguish different distance ranges between target obstacles and unmanned aerial vehicles, and can include D1, D2 and D3.

[0138] The prompt shape information can be a specific geometric shape used to represent the target obstacle, and can include mask shape, 3D mesh shape, and planar mesh shape, etc.

[0139] The color information can refer to the specific color assigned to the aforementioned specific geometric shape, which may include green, yellow, and red, etc.

[0140] The prompt shape and color information corresponding to different preset distance levels can be preset.

[0141] Based on the second position information of the target obstacle relative to the unmanned aerial vehicle (UAV) and a preset distance level, the method of determining the cue shape information and / or cue color information of the target obstacle in the rendered image can be adopted. This method can be used to determine the distance information between the target obstacle and the UAV based on the second position information of the target obstacle relative to the UAV, compare the distance information with each preset distance level, and adopt the cue shape information and / or cue color information corresponding to the preset distance level satisfied by the distance information.

[0142] For example, if the distance information is less than D1, the prompt shape is determined to be a mask shape and the prompt color is green; if the distance information is less than D2, the prompt shape is determined to be a 3D grid shape and the prompt color is yellow; if the distance information is less than D3, the prompt shape is determined to be a 2D grid shape and the prompt color is red.

[0143] Figure 10 This is an example image of a rendered image provided in Embodiment 4 of the present invention. Figure 10 As shown, the distance information of the ground is less than D1, so the prompt shape information of the ground adopts a mask shape. The distance information of the two trees behind is less than D2, so the prompt shape information of these two trees adopts a 3D grid shape. The distance information of the tree in front is less than D3, so the prompt shape information of this tree adopts a planar grid shape.

[0144] The advantage of this solution is that by determining the target obstacle's shape and / or color information in the rendered image based on the second position information of the target obstacle relative to the unmanned aerial vehicle and a preset distance level, the position, distance, and potential risk level of the obstacle can be displayed to the user intuitively and clearly, thereby improving the user's perception of the surrounding environment and effectively avoiding collision accidents.

[0145] In this technical solution, optionally, the rendering style of the target obstacle is determined based on the second position information of the target obstacle relative to the unmanned aerial vehicle, including:

[0146] If the target obstacle is outside the field of view of the image data, then based on the second position information of the target obstacle relative to the unmanned aerial vehicle and a preset distance level, the proximity prompt shape information, prompt color information and / or prompt distance information of the target obstacle in the rendered image are determined.

[0147] Preset distance levels can refer to pre-defined standard levels used to distinguish different distance ranges between target obstacles and unmanned aerial vehicles, and can include D4 and D5, etc.

[0148] Proximity cue shape information can be a specific geometric shape used to indicate that a target obstacle outside the field of view of the image data is approaching, and can include borders, etc.

[0149] The color information can refer to the specific color assigned to the aforementioned specific geometric shape, which may include green, yellow, and red, etc.

[0150] Distance information can refer to the proximity of a target obstacle outside the field of view of the image data.

[0151] Based on the second position information of the target obstacle relative to the unmanned aerial vehicle (UAV) and a preset distance level, the method of determining the cue shape information and / or cue color information of the target obstacle in the rendered image can be adopted. This method can be used to determine the distance information between the target obstacle and the UAV based on the second position information of the target obstacle relative to the UAV, compare the distance information with each preset distance level, and adopt the proximity cue shape information, cue color information and / or cue distance information corresponding to the preset distance level satisfied by the distance information.

[0152] For example, if the distance information is less than D4, then the shape information of the adjacent prompt is determined to be a border, the color information of the prompt is yellow, and the distance information of the prompt is the distance information. If the distance information is less than D5, then the shape information of the adjacent prompt is determined to be a border, the color information of the prompt is red, and the distance information of the prompt is the distance information.

[0153] like Figure 10 As shown, there is a target obstacle above and its distance information is 4 meters (less than D4), so the rendered image shows a yellow border and a distance information of 4 meters at the top. There is a target obstacle on the left and its distance information is 2 meters (less than D5), so the rendered image shows a red border and a distance information of 2 meters on the left.

[0154] The advantage of this scheme is that, even when the target obstacle is outside the field of view of the image data, based on the second position information of the target obstacle relative to the unmanned aerial vehicle and the preset distance level, the proximity prompt shape information, prompt color information and / or prompt distance information of the target obstacle in the rendered image can be determined. This allows users to intuitively and clearly understand the position, distance and potential risk level of target obstacles outside the field of view of the image data, thereby improving the user's perception of the surrounding environment and effectively avoiding collision accidents.

[0155] S906. Based on the first position information of the target obstacle in the image data, render the target obstacle to obtain a rendered image.

[0156] The advantage of this solution is that by determining the rendering style of the target obstacle based on its second position information relative to the unmanned aerial vehicle (UAV), users can quickly judge the degree of danger of the target obstacle when viewing the rendered image, and thus adjust the flight path of the UAV or other corresponding obstacle avoidance measures in a timely manner.

[0157] Example 5

[0158] Figure 11 This is a schematic diagram of the obstacle rendering device provided in Embodiment 5 of this application. Figure 11 As shown, it specifically includes the following:

[0159] Data acquisition module 1110 is used to acquire image data and point cloud data collected by unmanned aerial vehicles;

[0160] The first position determination module 1120 is used to identify the semantic information of the image data and determine the first position information of the obstacle in the image data based on the semantic information.

[0161] The second position determination module 1130 is used to determine the second position information of the obstacle relative to the unmanned aerial vehicle based on the point cloud data.

[0162] The image rendering module 1140 is used to render the obstacle in the image data based on the first position information and the second position information according to the preset rendering rules, so as to obtain a rendered image.

[0163] In this embodiment of the invention, a data acquisition module is used to acquire image data and point cloud data collected by the unmanned aerial vehicle (UAV); a first position determination module is used to identify the semantic information of the image data and determine the first position information of the obstacle in the image data based on the semantic information; a second position determination module is used to determine the second position information of the obstacle relative to the UAV based on the point cloud data; and an image rendering module is used to render the obstacle in the image data based on the first and second position information according to a preset rendering rule to obtain a rendered image. This obstacle rendering device, by rendering obstacles in the image data according to preset rendering rules, can meet the obstacle avoidance and early warning needs of UAVs with limited computing power, rationally allocate obstacle rendering resources, thereby assisting users in safe flight control, and is applicable to various UAVs.

[0164] The obstacle rendering apparatus provided in this application embodiment has the same beneficial effects as the obstacle rendering method provided in the above embodiments, and will not be described again here to avoid repetition.

[0165] Example 6

[0166] Figure 12 This is a structural schematic diagram of an unmanned aerial vehicle provided in Embodiment Six of the present invention. Figure 12 As shown in the illustration, this application also provides an unmanned aerial vehicle (UAV) device 1200, which includes: one or more processors 1201; and a storage device 1202 for storing one or more programs. The one or more programs can be executed by the one or more processors 1201, which can implement the various processes of the obstacle rendering method embodiments described above and achieve the same technical effects. To avoid repetition, further details are omitted here.

[0167] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0168] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile device, mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0169] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for rendering obstacles, characterized in that, The method is performed by an unmanned aerial vehicle, and the method includes: Acquire image data and point cloud data collected by unmanned aerial vehicles; Identify the semantic information of the image data, and determine the first position information of the obstacle in the image data based on the semantic information; The second position information of the obstacle relative to the unmanned aerial vehicle is determined based on the point cloud data; According to preset rendering rules, the obstacle is rendered in the image data based on the first position information and the second position information to obtain a rendered image.

2. The obstacle rendering method according to claim 1, characterized in that, After obtaining the rendered image, the method further includes: The rendered image is sent to the user control terminal for display on the user control terminal.

3. The obstacle rendering method according to claim 1, characterized in that, According to preset rendering rules, the obstacle is rendered in the image data based on the first position information and the second position information to obtain a rendered image, including: The target obstacle located within the preset rendering area is determined based on the second location information; Based on the first position information of the target obstacle in the image data, the target obstacle is rendered to obtain a rendered image.

4. The obstacle rendering method according to claim 3, characterized in that, Based on the first position information of the target obstacle in the image data, the target obstacle is rendered to obtain a rendered image, including: Obtain the directional intervals obtained by pre-dividing the preset rendering area, and determine the preset rendering weight of each directional interval; Based on the first position information of the target obstacle in the image data, the target obstacle is rendered according to the preset rendering weight of the azimuth interval where the target obstacle is located, and a rendered image is obtained.

5. The obstacle rendering method according to claim 4, characterized in that, Obtaining the pre-divided directional intervals of the preset rendering region, and determining the preset rendering weights of each directional interval, including: Obtain the directional intervals obtained by pre-dividing the preset rendering area, wherein the division method includes at least one of the following: division according to the distance in front of the flight, division according to the distance above and below the flight, and division according to the distance to the left and right of the flight. Obtain the weight components of each directional interval under each division method; Based on the weight components of the current directional interval under each division method, determine the preset rendering weight of the current directional interval, and traverse each directional interval to obtain the preset rendering weight of each directional interval.

6. The obstacle rendering method according to claim 4, characterized in that, Based on the first position information of the target obstacle in the image data, the target obstacle is rendered according to a preset rendering weight of the orientation interval where the target obstacle is located, to obtain a rendered image, including: Obtain the number of elements displayed by the unmanned aerial vehicle; Calculate the sum of the preset rendering weights for the location range of each target obstacle; Divide the target number of elements to be displayed by the total to obtain the number of elements to be displayed per unit rendering weight; For each target obstacle, the preset rendering weight of the directional interval where the target obstacle is located is multiplied by the number of elements displayed corresponding to the unit rendering weight to obtain the number of target elements displayed for the target obstacle. For each target obstacle, based on the first position information of the target obstacle in the image data, the target obstacle is rendered according to the number of target elements displayed to obtain a rendered image.

7. The obstacle rendering method according to claim 3, characterized in that, According to preset rendering rules, the obstacle is rendered in the image data based on the first position information and the second position information to obtain a rendered image, and the method further includes: Based on the second location information, identify whether there are obstacles within the preset warning area; If an obstacle is identified as being within a preset warning area, a warning message is rendered, resulting in a rendered image.

8. The obstacle rendering method according to claim 3, characterized in that, After determining the target obstacle located within the preset rendering area based on the second location information, the method further includes: The rendering style of the target obstacle is determined based on the second position information of the target obstacle relative to the unmanned aerial vehicle.

9. The obstacle rendering method according to claim 8, characterized in that, Based on the second position information of the target obstacle relative to the unmanned aerial vehicle, the rendering style of the target obstacle is determined, including: If the target obstacle is within the field of view of the image data, then based on the second position information of the target obstacle relative to the unmanned aerial vehicle and a preset distance level, the prompt shape information and / or prompt color information of the target obstacle in the rendered image are determined.

10. The obstacle rendering method according to claim 8, characterized in that, Based on the second position information of the target obstacle relative to the unmanned aerial vehicle, the rendering style of the target obstacle is determined, including: If the target obstacle is outside the field of view of the image data, then based on the second position information of the target obstacle relative to the unmanned aerial vehicle and a preset distance level, the proximity prompt shape information, prompt color information and / or prompt distance information of the target obstacle in the rendered image are determined.

11. An obstacle rendering device, characterized in that, The device includes: The data acquisition module is used to acquire image data and point cloud data collected by the unmanned aerial vehicle. The first position determination module is used to identify the semantic information of the image data and determine the first position information of the obstacle in the image data based on the semantic information. The second position determination module is used to determine the second position information of the obstacle relative to the unmanned aerial vehicle based on the point cloud data; The image rendering module is used to render the obstacle in the image data based on the first position information and the second position information according to the preset rendering rules, so as to obtain a rendered image.

12. An unmanned aerial vehicle, characterized in that, The unmanned aerial vehicle includes: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the steps of the obstacle rendering method according to any one of claims 1-10.

13. A storage medium for storing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the steps of the obstacle rendering method according to any one of claims 1-10.

Images