Calculating lateral distance from uncrewed autonomous vehicle to object

Abstract

A communication device (100) being or associated with an Uncrewed Autonomous Vehicle, UAV, (110). The communication device is configured to detect at least one object (140) from at least one image or video and calculate at least one lateral distance from the UAV to the detected object. It further is configured to compare the lateral distance with a determined value times a current altitude of the UAV and issue a message to a receiving unit (150) if the lateral distance is less or equal to the determined value times the altitude of the UAV. A method, a computer program (710), a computer program product (770) and a computer-readable storage medium (730) are also disclosed.

Description

[0001] CALCULATING LATERAL DISTANCE FROM UNCREWED AUTONOMOUS

[0002] VEHICLE TO OBJECT

[0003] TECHNICAL FIELD

[0004] The disclosure relates to a communication device being or associated with an Uncrewed Autonomous Vehicle (UAV) and a method performed by a communication device. A related computer program, computer program product and computer- readable storage medium are also disclosed.

[0005] BACKGROUND

[0006] A flight with a UAV can involve hovering in the air, high and low speeds, and rapid changes in direction. A remote pilot controlling the UAV must maintain spatial awareness in regard to the ground for safety reasons, even if the environment around the UAV is changing rapidly. Any person not involved in the flight of the UAV is generally not responsible for controlling the distance to the UAV. Thus, the remote pilot must maintain a distance from the uninvolved persons on the ground to ensure the safety of the uninvolved persons.

[0007] Regulations have therefore been introduced in some geographical areas, such as the European Union. The regulations may comprise a safety rule that the remote pilot should keep the UAV at least at a lateral distance from an uninvolved person. The lateral distance should not be less than the altitude of the UAV. That is, if the UAV is flying at a height of 5 meters, for example, a horizontal distance from an uninvolved person to the UAV should be at least 5 meters. This rule is often referred to as the 1 :1 rule.

[0008] This rule is in practice difficult to follow, because e.g., it may be hard for a remote pilot to accurately estimate the distances between the UAV and other objects, such as a lateral distance between the UAV and other objects, or altitude of the UAV without further expertise or experience to follow this rule. In addition, even if the remote pilot is experienced, he or she might not accurately estimate the distances, which could result in an incorrect estimate that risks the safety of uninvolved persons. To comply with the rule and to keep enough safety distance, the remote pilot generally must also consider his reaction time. For example, he or she must determine a braking distance to check the UAV's maneuverability, which is the time and flight distance it takes for the UAV to come to a complete stop. The braking distance depends on the UAV's size, mass, and aerodynamic drag. In addition, a remote pilot who controls a UAV and who is not in the vicinity to the UAV would not easily be able to comply with safety regulations. This may apply also for UAVs with preconfigured flight paths. Thus, the attempt to maintain a sufficiently safe distance from the uninvolved person and to comply with the 1 :1 rule could fail due to faulty perception of the UAV’s maneuverability, resulting in a fatal risk to the safety of persons in the vicinity of the UAV.

[0009] There have been some attempts to create an improved control for monitoring the environment of the UAV. For example, US 20210004003 A1 discloses a system for an artificial intelligence (Al) controller, which may pilot a UAV based on various inputs, including sensor inputs and / or inputs from multiple cameras, to detect and locate features in its environment. This enables in-depth field information that can be extracted from a computer vision (CV) unit. However, the presented experiments do not provide a solution on how to comply with the rule that the lateral distance to an uninvolved person must not be less than the UAVs flight altitude. In detail, no solution is provided on how to maintain a sufficiently safe distance to uninvolved persons, who might be moving, and how to accurately determine the distances for safety reasons.

[0010] SUMMARY

[0011] An object of the invention is to enable safer operation of a UAV.

[0012] A first aspect of the invention relates to a communication device being or associated with an UAV, wherein the UAV comprises a camera arrangement, comprising at least one camera for creating at least one image or at least one video, and the communication device comprises a processing circuitry and a memory, which comprises instructions executable by the processing circuitry. The instructions, when executed by the processing circuitry, cause the communication device to be configured to detect at least one object from the image or video; calculate at least one lateral distance from the UAV to the detected object; compare the lateral distance with a determined value times a current altitude of the UAV; and issue a message to a receiving unit if the lateral distance is less or equal to the determined value times the altitude of the UAV. Hereby is achieved that a message is issued if the lateral distance is less or equal to the determined value times the altitude of the UAV.

[0013] According to an embodiment of the first aspect the camera arrangement comprises a gimbal for rotation of the camera arrangement. Hereby is achieved that the camera arrangement may be rotatable for correct or at least improved camera direction in relation to an object.

[0014] According to an embodiment of the first aspect the processing circuitry make the communication device to be configured to generate a bounding box around a projection of the detected object on an image plane of the image or video; and classify the detected object into a class representing a person, a building, a vehicle, or a class representing any other item than a person, a building, and a vehicle. Hereby is achieved that the detected object may be identified as a certain type of object which then can be used to determine whether the 1 :1 rule must be followed with respect to that object. In an embodiment the calculation of the lateral distance from the UAV to the detected object is based on a focal length of the camera arrangement, a gimbal pitch of the camera arrangement, a height of the bounding box, a height of the detected object and a fraction of the upper part of the bounding box of the respective image plane, wherein the upper part is the part of the bounding box facing away from a ground on which the detected object is located. Hereby is achieved that the lateral distance may be determined for further processing. In an embodiment the calculation of the lateral distance from the UAV to the detected object is based on wherein d|_| is the lateral distance from the UAV to the detected object, Hj is the height of the bounding box, 0 is the gimbal pitch of the camera arrangement, f is the focal length of the camera arrangement, HR is the height of the detected object and x is the fraction of the upper part of the bounding box of the image plane. Hereby is achieved that the lateral distance may be determined for further processing. In an embodiment the focal length of the camera arrangement can be extracted from a focal length field of photo metadata of the image or the video or calculated using an image width of the image or video and a sensor width of the camera arrangement. Hereby is achieved that the focal length may be determined of the camera arrangement for more precise lateral distance calculation. In an embodiment the fraction x equals 2 if the bounding box is centered vertically to the image plane by adjusting the gimbal of the camera arrangement. Hereby is achieved that the lateral distance may be calculated after adjusting the gimbal of the camera arrangement.

[0015] According to an embodiment of the first aspect the processing circuity causes the communication device to be configured to select the shortest lateral distance if two or more objects are detected. Hereby is achieved that the lateral distance to the closest detected object may be selected. In an embodiment the lateral distance is the shortest lateral distance. Hereby is achieved that the shortest lateral distance to the closest object may be further processed.

[0016] According to an embodiment of the first aspect the determined value is any number equal or greater than one. Hereby is achieved that irregularities for comparison to the current altitude of the UAV may be considered.

[0017] According to an embodiment of the first aspect the value is determined based on uncertainty data. Hereby is achieved that the value may be determined based on uncertainties.

[0018] According to an embodiment of the first aspect the processing circuity causes the communication device to be configured to prevent the UAV to move further in the direction of the detected object if the lateral distance is less or equal to the determined value times the altitude of the UAV. Hereby is achieved that movements towards the detected objects may be avoided. In an embodiment the prevention of the UAV to move further in the direction of the detected object further approaches in the direction of the detected object with the smallest lateral is initiated by the receiving unit. Hereby is enabled that the receiving unit may be able to initiate the prevention.

[0019] According to an embodiment of the first aspect the communication device is the UAV and the receiving unit comprised in a device handled by a user or a controller device or an air traffic management facility. Hereby is enabled that the receiving unit may be able to control the UAV. According to an embodiment of the first aspect the communication device is external from the UAV, and the receiving unit comprises a device handled by a user or a controller device or an air traffic management facility or the UAV. Hereby is enabled that the receiving unit may be able to control the UAV.

[0020] According to an embodiment of the first aspect the issue of a message to the receiving unit comprises signaling over a short-range radio network or over a 3rd Generation Partnership Project, 3GPP, wireless network. Hereby is achieved that the UAV may be controlled via wireless signaling.

[0021] According to an embodiment of the first aspect multispectral object detection is enabled by combining images or videos from one or more cameras in the camera arrangement. Hereby is achieved that multispectral object detection may be applicable.

[0022] According to an embodiment of the first aspect the calculation of the lateral distance from the UAV to the detected object is only done if the detected object is classified to be a person. Hereby is achieved that verification may be complied with if there is a safety risk.

[0023] According to an embodiment of the first aspect the height of the detected object is approximated by an average human height or estimated using a stereo-type arrangement, wherein the average human height is between 100 centimeter and 250 centimeter. Hereby is achieved that the lateral distance may be calculated based on the height of the detected object.

[0024] According to an embodiment of the first aspect the camera is a monocular camera.

[0025] According to an embodiment of the first aspect the camera arrangement comprises a first visual camera, and either a thermal camera or a second visual camera. Hereby is achieved that the verification may be based on different cameras.

[0026] According to an embodiment of the first aspect the communication device is associated with the UAV and is a network device. Hereby is achieved that communication between the UAV and the communication device may be enabled. According to an embodiment of the first aspect the communication device is associated with the UAV and is an end user device. Hereby is achieved that the UAV may be controllable by a user.

[0027] A second aspect of the invention related to a method performed by a communication device being or associated with an Uncrewed Autonomous Vehicle (UAV), wherein the UAV comprises a camera arrangement, comprising at least one camera for creating at least one image or video. The method comprises detecting at least one object from the image or video; calculating at least one lateral distance from the UAV to the detected object; comparing the lateral distance with a determined value times a current altitude of the UAV; and issuing a message to a receiving unit if the lateral distance is less or equal to a determined value times the altitude of the UAV. Hereby is achieved that a message is sent if the lateral distance is less or equal to the determined value times the altitude of the UAV.

[0028] According to an embodiment of the second aspect the camera arrangement comprises a gimbal for rotation of the camera arrangement.

[0029] According to an embodiment of the second aspect, the method comprises generating a bounding box around a projection of the detected object on an image plane of the image or video; and classifying the detected object into a class representing a person, a building, a vehicle, or a class representing any other item than a person, a building, and a vehicle. In an embodiment calculating the lateral distance from the UAV to the detected object is based on a focal length of the camera arrangement, a gimbal pitch of the camera arrangement, a height of the bounding box, a height of the detected object and a fraction of the upper part of the bounding box of the respective image plane, wherein the upper part is the part of the bounding box facing away from a ground on which the detected object is located. In an embodiment of the method calculating the lateral distance from the UAV to the detected object is based on wherein d|_| is the lateral distance from the UAV to the detected object, Hj is the height of the bounding box, 0 is the gimbal pitch of the camera arrangement, f is the focal length of the camera arrangement, HR is the height of the detected object and x is the fraction of the upper part of the bounding box of the image plane. In an embodiment the focal length of the camera arrangement can be extracted from a focal length field of photo metadata of the image or the video or calculated using an image width of the image or video and a sensor width of the camera arrangement. In an embodiment the fraction x equals 2 if the bounding box is centered vertically to the image plane by adjusting the gimbal pitch of the camera arrangement.

[0030] According to an embodiment of the second aspect, the method comprises selecting the shortest lateral distance if two or more objects are detected. In an embodiment of the method the lateral distance is the shortest lateral distance.

[0031] According to an embodiment of the second aspect, the determined value is any number equal or greater than one.

[0032] According to an embodiment of the second aspect, the value is determined based on uncertainty data

[0033] According to an embodiment of the second aspect, the method comprises preventing the UAV to move further in the direction of the detected object if the lateral distance is less or equal to the determined value times the altitude of the UAV. In an embodiment preventing the UAV to move further in the direction of the detected object is initiated by the receiving unit.

[0034] According to an embodiment of the second aspect, the communication device is the UAV and the receiving unit comprised in a device handled by a user or a controller device or an air traffic management facility.

[0035] According to an embodiment of the second aspect, the communication device is external from the UAV, and the receiving unit comprises a device handled by a user or a controller device or an air traffic management facility or the UAV.

[0036] According to an embodiment of the second aspect, issuing a message to the receiving unit comprises signaling over a short-range radio network or over a 3rd Generation Partnership Project, 3GPP, wireless network.

[0037] According to an embodiment of the second aspect, multispectral object detection is enabled by combining images or videos from one or more cameras in the camera arrangement. According to an embodiment of the second aspect, calculating the lateral distance from the UAV to the detected object is done if the detected object is classified to be a person.

[0038] According to an embodiment of the second aspect, the height of the detected object is approximated by an average human height or estimated using a stereo-type arrangement, wherein the average human height is between 100 centimeter and 250 centimeter.

[0039] According to an embodiment of the second aspect, the at least one camera is a monocular camera.

[0040] According to an embodiment of the second aspect, the camera arrangement comprises a first visual camera, and either a thermal camera or a second visual camera. Hereby is achieved that the verification may be based on different cameras.

[0041] According to an embodiment of the second aspect, the communication device associated with the UAV and is a network device. Hereby is achieved that communication between the UAV and the communication device may be enabled.

[0042] According to an embodiment of the second aspect, the communication device associated with the UAV and is an end user device. Hereby is achieved that the UAV may be controllable by a user.

[0043] According to a third aspect of the invention, there is a computer program product comprising a computer readable storage medium. The computer readable storage medium has computer readable code embodied therein, the computer readable code being configured such that, on execution by a communication device being or associated with an Uncrewed Autonomous Vehicle, UAV, the communication device is caused to perform a method according to the second aspect or any embodiment of the second aspect.

[0044] According to a fourth aspect of the invention, there is a computer program comprising a computer readable code which, when run on a communication device being or associated with an Uncrewed Autonomous Vehicle (UAV) causes the communication device to perform the method according to the second aspect or any embodiment of the second aspect. According to a fifth aspect of the invention, there is a computer-readable storage medium comprising a computer program according to the fourth aspect.

[0045] BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figures 1 shows a schematic diagram illustrating an example of an environment in which embodiments presented herein can be applied.

[0047] Figure 2 is a flowchart illustrating a method performed by the communication device.

[0048] Figure 3 is a flowchart of embodiments of the method.

[0049] Figure 4 illustrates a geometry of an environment in which embodiments presented herein can be applied.

[0050] Figure 5 illustrates an exemplary uncrewed autonomous vehicle according to an embodiment of the invention.

[0051] Figure 6 illustrates a schematic diagram illustrating an example of an environment in which embodiments presented herein can be applied.

[0052] Figure 7 illustrates an exemplary communication device according to an embodiment of the invention.

[0053] DETAILED DESCRIPTION

[0054] Figure 1 illustrates a schematic diagram illustrating an example of an environment in which embodiments presented herein can be applied. It relates to a communication device 100 being or associated with an UAV 110. The UAV comprises a camera arrangement 120, comprising at least one camera 130 for creating at least one image or at least one video. Figure 1 also illustrates an object 140, in the Figure indicated as a human, and a receiving unit 150, wherein the receiving unit is optionally comprised in a device handled by a user 160. The receiving unit 150 is optionally the UAV 110.

[0055] Figure 1 is further illustrating the concept of the 1 :1 rule, which can be applied for safety reasons. To satisfy the rule, a lateral distance of the UAV to, e.g., an uninvolved person is greater than the flight altitude of the UAV or greater than a determined value times the flight altitude of the UAV. In other words, the UAV performing, for example, a flight survey must not move closer to for example the uninvolved person in the vertical distance or a value times the vertical distance than its flight altitude. The value must in the case of the 1 :1 rule be at least 1 , but is alternatively, e.g. 1.1 or 1.2 to introduce a safety margin. Illustrated are the uninvolved person on the ground and the UAV above the ground, which is for example hovering or moving along a flight path. The lateral distance is d |_| and denotes the horizontal distance of the UAV to the person. The altitude of the UAV is dvand denotes the flying height of the UAV, thus the vertical distance to the person.

[0056] Figure 2 is a flowchart illustrating a method performed by the communication device 100, which either is the UAV 110, or is associated with the UAV. The method comprises a first step 210 which is detecting the at least one object 140 from the image or video. The method further comprises a calculation step 220, wherein at least one lateral distanced^ from the UAV to the detected object is calculated. In a comparing step 230 at least one lateral distance is compared with a determined value times the current altitude of the UAV. By comparing the lateral distance with the altitude, the 1 :1 rule or variant thereof may be verified. In an issuing step 240 a message is issued to the receiving unit 150 if the lateral distance is less or equal to the determined value times the altitude d of the UAV. In some embodiments, the message may be, for example, a warning to a remote pilot via the receiving unit or a command indicating an action to correct a flight path of the UAV to comply with the rule.

[0057] Advantages are provided by the invention or any of its embodiments are, for example, step 210 may address the challenges described above of faulty perception, by enabling a non-human intervention for detecting objects on the ground. The object could be, for example, classified as a person. This could advantageously be used to monitor the environment and improve spatial awareness of, for example, a remote pilot. The detection of an object may enable the calculation of the lateral distance. The calculation in step 220 of the lateral distance based on the image or video of the environment of the UAV is provided to address the challenge described above of maintaining a sufficient safe distance to, for example, an uninvolved person and to estimate the distances, the lateral distance and the height distance to, for example, the uninvolved person, more accurately than, for example, using eyesight. The calculation of the lateral distance may thus allow a more accurate verification of compliance with the 1 :1 rule. Advantageously, the lateral distance can be calculated without the influence of ground unevenness, which can enable a more accurate determination of the distance. The value in the issuing step 240 could for example address the challenge described above of taking uncertainties as breaking distance, or environmental influences into account, which could lead, for example, to a better assessment of the UAV’s maneuverability. The value could be for example 1 .0 for the 1 :1 rule, or 1.1 to provide a 10% margin before the 1 :1 rule would be violated. The value may be dynamically set in dependence of, e.g., the speed of the LIAV or the environmental influences. Thus, it may be periodically updated. The issued message may enable, for example, precise control of the safe distance to uninvolved persons. The action can be advantageous because it enables more safety, for example, in the case of a pre-configured path during a flight. The invention may lead to more safety, for example, when the remote pilot is not in the vicinity. It may allow for better controllability and reliability in following the rule and keeping a sufficient safety distance to uninvolved persons.

[0058] Figure 3 is a flowchart illustrating embodiments of the method, which embodiments are used to verify compliance with the 1 :1 rule.

[0059] In an embodiment the method steps 310 to 330 could be, for example, performed by an object detector in the form of a visual object detector. Examples of a suitable visual object detector are deep convolutional networks, such as Faster Regionbased Convolutional Neural Networks (R-CNN) or regression-based object detectors as You Only Look Once (YOLO) type detectors. Other object detectors, such as R- CNN, Fast R-CNN or Single-shot Detector (SSD), are also appropriate.

[0060] In a step 310, at least one object 140 from the image or video is detected. The detected object could be for example a person, a building, a vehicle or any other item. Any other item could be for example any item in a surrounding environment in which the UAV 110 is located, such as a tree, a mountain, a wind turbine, a street light, or a high-voltage line. In an example of using a deep convolutional network based visual object detector, the image or video frames, where the video frames may be extracted from the video, could be provided as input to a convolutional network that provides a convolutional feature map. A separate convolutional network comprised in the visual object detector could be used to predict region suggestions. In another example of the use of a regression-based visual object detector, the image or video frames, where the video frames can be extracted from the video, is processed in parts by a network. The image or frame could be divided into an n-by-n grid, where n is the number of squares in the horizontal or vertical direction.

[0061] An optional step 320 comprises generating a bounding box around a projection of the detected object on an image plane of the image or video. A two-dimensional (2D) image of the object is projected on an image plane. The projection of the detected object could lie in the image plane of the camera arrangement 120. The image plane is the plane through the focus perpendicular to the axis of the at least one lens. The bounding box is a digital, box, or a digital representation thereof, that outlines the detected object in an image. A frame of the box encloses the detected object projection in the image plane. The frame is, for example, rectangular, i.e. the bounding box would in the case of a rectangular frame also be rectangular. In another example the bounding box has a per-pixel segmentation which comprises a silhouette of the object. The bounding box can, for example, specify the position, or at least an indication of the position, of the detected object, and comprise data about the class of the object and / or confidence value which indicates the degree of probability that the object is present in the bounding box. Metadata of the bounding box may comprise at least coordinates of the bounding box within the image and associated labels of the detected object. The class and confidence value are concrete representations of semantic information of the associated labels. The bounding box is, for example, an output of the visual object detector by creating a frame or a per-pixel segmentation around the detected object projection in the image plane. Some visual object detectors may output per-pixel segmentation instead of the bounding box. The frame is created by considering the smallest coordinate values in the image plane that still contain the detected object projection. In the example of using a deep convolutional network for visual object detection, the proposed region is reshaped using a Region of Interest (Rol) pooling layer that is used to predict the offset values for the bounding box. In the other example of using a regression-based visual object detector, the network could output offset values for a bounding box for each square. An optional step 330 comprises classifying the detected object into one or more of the classes: a first class representing a person, a second class representing a building, a third class representing a vehicle, and a fourth class representing any other item than a person, a building, and a vehicle. Any other item could be for example any item in a surrounding environment the UAV is located in, for example a tree, a mountain, a wind turbine, a street light, or a high-voltage line. The class represents a membership of an object contained in an image.

[0062] Classification could be based on image classification for object detection, which could be included in visual object detection. Image classification refers to assigning a label or class to an entire image. An image classification model takes the image as input for classification. The image classification model is trained for image classification. It analyzes and categorizes classes of images. In the example of using a deep convolutive network based visual object detector, the reshaped region is used to classify the image. In the other example of using a regression-based visual object detector, the class probability for the bounding box is returned. The bounding box with a class probability above a certain threshold is used.

[0063] In an embodiment multispectral object detection is enabled by combining images or videos from one or more cameras 130 in the camera arrangement. The multispectral object detection may enable a better performance of detecting objects in an image or video by collecting a multispectral image dataset for the object.

[0064] A step 340 comprises calculating at least one lateral distance d |_| from the UAV to the detected object. The lateral distance is defined by the horizontal distance from the UAV to the detected object. In case there are more than one objects, the step comprises calculating more than one lateral distances to the respective objects.

[0065] The optional step 340 is performed in case the detected object is classified to be a person. In other words, within verifying compliance with the 1 :1 rule, the classification of the detected object can be used to identify an uninvolved person. For example, all detected persons are in one embodiment first tagged as uninvolved. An involved person may in such an embodiment be identified based on the person’s vicinity to the UAV’s initial position and return position (e.g., a home / start point of the UAV). In a further example, a user may tag an object as being an involved person using for example an AprilTag or a Quick Response (QR) code in the environment of the UAV for identifying the involved person. In case one or more objects are classified as one or more persons, step 330 may comprise identifying one or more involved person and tagging all person as uninvolved, if they are not identified / tagged as involved. The uninvolved person could be any person not involved in the flight of the UAV. For example, the uninvolved person may be a passerby, a resident in the environment in which the UAV is flying, or any other person who is unaware of the UAV or in any way not associated with the flying of the UAV. The embodiment enables a more selective approach, as the safety restriction apply on persons, particularly uninvolved persons.

[0066] The height is, for example, approximated by an average human height or estimated using a camera arrangement 120 in the form of a stereo-type camera arrangement 121 , wherein the average human height is between 100 centimeters and 250 centimeters. The average human height could vary dependent on where the UAV is operated. For example, it may depend on average statistical human height data based on the country the UAV is operated in. It enables a simple solution for calculating the lateral distance. In case no average human height is used, distance measurements may be performed to measure a length of a first vector to determine a distance between the UAV and the top of the person, a second vector to determine a distance from the UAV to the bottom of the person and the angle between the first and the second vector. Using trigonometry, the human height may be calculated. The distance measurements may be performed by a distance measurement tool, such as laser distance meters or ultrasonic sensors. In a further example the human height may be estimated using the stereo-type camera arrangement. The stereotype camera arrangement may be the camera arrangement 120 comprising at least one stereo camera in the camera arrangement. Based on the base of the stereo camera, i.e. , the distance between at least two cameras in the stereo camera, the stereo camera enables triangulation and thus depth determination in an image scene. Based on the base of the stereo camera and the triangulation, the human height can be estimated.

[0067] The calculation of the lateral distance from the UAV to the detected object in step 340 is, for example, based on the focal length of the camera arrangement, the gimbal pitch of the camera arrangement, a height of the bounding box, the height of the detected object and a fraction of the upper part of the bounding box of the respective image plane, wherein the upper part is the part of the bounding box facing away from a ground on which the detected object is located. The fraction is the value by which the detected object deviates vertically from the center of the bounding box. The bounding box can be therefore divided in an upper and lower part based on this deviation.

[0068] In an embodiment, the fraction x equals 2, if the bounding box centered vertically to the image plane of the detected object projection by adjusting the gimbal of the camera arrangement. In an example, the adjustment of the gimbal can be comprised in detecting at least one object from the image or video, in step 310.

[0069] In an embodiment, the calculation of the lateral distance from the UAV to detected object in step 340 is based on wherein d|_| is the lateral distance from the UAV to the detected object, Hj is the height of the bounding box, 6 is the gimbal pitch of the camera arrangement, f is the focal length of the camera arrangement, HR is the height of the detected object and x is the fraction of the upper part of the bounding box of the image plane.

[0070] Figure 4 illustrates a geometry of the environment for deriving the equation above as a non-limiting example. Out of trigonometry the following is given:

[0071] ZMB=ZMZHsin 0, and (Eq. 2)

[0072] BZH=ZMZHCOS 0' (Eq. 3) wherein Z^ is the coordinate indicating the height of the detected object HR, Z|y| is

[0073] HR the coordinate indicating half of the height of the detected object — and B indicates the coordinate point of the incident perpendicular drawn on a straight line from a UAV coordinate D to ZR: DZR to a straight line from the coordinate D to Z^: DZ^, wherein the bar indicates the straight line between the coordinates.

[0074] The intercept theorem gives: (Eq. 4) wherein F is the coordinate point of the focal length f on DZ and E denotes the point on DZ|_|, when drawing the incident perpendicular on the point F.

[0075] Combined with the equation above, Eq. 6, follows:

[0076] The total distance from the object fraction to the camera origin is given by:

[0077] ZMD=ZMB+BD. (Eq. 6)

[0078] This results in:

[0079] Taking a straight line from the camera position D to the point K denoting the horizontal distance between the detected object and the UAV, into account, trigonometry results in:

[0080] KD=ZMD COS 0 . (Eq. 8)

[0081] Together with the equation above it results in:

[0082] KD denotes the lateral distance d|_| . Z^ Z|_| is half of the height of the detected object

[0083] Ho — —

[0084] — . DF is the focal length f and FE stands for the height of the bounding box Hj, divided by the fraction x. That results in:

[0085] Other ways to obtain the lateral distance are also appropriate. The advantage of using the above-described way on calculating the lateral distance is that no distance measurements with a distance measurement tool, such as laser distance meters are necessary. Referring again to figure 3, an optional method step 350 comprises selecting the shortest lateral distance if two or more objects are detected. The selection of the shortest lateral distance can be done, for example, by sorting one or more lateral distances according to their value. Two or more objects could be detected, when, for example, two or more uninvolved persons are within detection range from the UAV. In case of two or more detected objects, such as two or more objects within the same class, for example the first class mentioned above, the steps 320 and 330 could be executed for each detected object. Out of the calculated lateral distances of each detected objects, the shortest distance is selected for verifying compliance the 1 :1 rule. This ensures that the rule that the lateral distance is less or equal to the determined value times the altitude of the UAV is observed for the person closest to the UAV. After verifying compliance with the 1 :1 rule for the closest person, the steps can be performed again to verify compliance with the rule based on other detected objects in the UAV's detection area.

[0086] The lateral distance used in a next step 360 is, for example, the shortest lateral distance. In other words, if two or more objects are detected, several lateral distances can be calculated, but only the shortest lateral distance may be processed further and the lateral distance values that are not the shortest may be deleted.

[0087] A step 360 comprises comparing the lateral distance with a determined value times the current altitude of the UAV. The lateral distance could be, for example, the shortest lateral distance if two or more objects are detected as mentioned above. The determined value may be any number equal or great than one. The value could for example, lie in between 1 and 1 .5. The value may be determined, for example, based on uncertainty data or based on a predefined value. Likely uncertainties may be for example at least one of the following uncertainties: reaction time of a pilot of the UAV, a breaking time for the UAV, processing time or detection uncertainties. Detection uncertainties may be at least one of uncertainties arising due to movement of the detected object, incorrect classification, uncertainty in determining the fraction x, unprecise orientation of the bounding box or the gimbal pitch or blurredness of the image or video. The value may be determined either prior to performing method 300 and / or for example during step 360. The value may be also updated periodically during performing method 300, for example every second or every 10 seconds. The uncertainty data could be for example determined based on measurements of for example weather influences, poor light conditions, velocity of the detected object, taken at various additional sensors as for example a distance measurement tool or an anemometer. For example, the breaking time data could be determined by measuring periodically or continuously the wind velocity. The value also might be determined based on current uncertainty data and / or based on historic uncertainty data. Thus, the value could be for example determined based on the breaking time, which in turn is determined based on, for example, current uncertainty data and / or based on processing time, which in turn is determined based on, for example, historic measurements of processing time uncertainties.

[0088] The current altitude may be the last received altitude value received from at least one altimeter. The altimeter may be configured to measure the altitude periodically based on, for example, infrared measurements taken by a downward-facing sensor with which the UAV may be equipped, a barometer measurement to obtain altitude by measuring air pressure or receiving altitudes using triangulated data from a satellite-based positioning system, such as Global Positioning System (GPS), Galileo, BeiDou, and GLONASS), or satellite-based internet services like Starlink™. The UAV could also be equipped with multiple sensors to measure altitude based on different methods on a regular basis for more accurate determination, or it could also alternate between different methods for determination based on the environmental conditions in which the UAV is located.

[0089] A step 370 comprises issuing a message to a receiving unit 150 if the lateral distance is less or equal to the determined value times the altitude of the UAV. Thus the message is issued in case of non-compliance, or future risk of non-compliance with the 1 :1 rule. Thus, optionally, in one embodiment no message may be issued in the event of compliance with the 1 :1 rule. The message could be for example a Hypertext Transfer Protocol (HTTP) message such as a HTTP post message or a Short Message Peer-to-Peer Protocol (SMPP) message. The message could be, for example, a warning or a trigger, such as a message directed to a controller of the UAV or a command to initiate step 330. Step 330 could be initiated by the receiving unit.

[0090] The receiving unit may be comprised in a device handled by a user 160 or a controller device or an air traffic management facility if the communication device 100 is being the UAV. In a further example receiving unit may be comprised within the UAV, wherein the communication device is the UAV. The receiving unit could in this case be, for example, an internal flight control.

[0091] In another example the receiving unit comprises / is a device handled by a user or a controller device or an air traffic management facility or the UAV in case the communication device 100 is associated with the UAV. In an example, the receiving unit is the communication device 100 if the communication device 100 is associated with the UAV.

[0092] In the example of the receiving unit being the communication device, the message is issued within the communication device. Issuing a message within the communication device may be issuing a trigger or a command for triggering the communication device to initiate step 380. The trigger or command may be issued using, e.g., an application layer protocol as an open web socket protocol or HTTP. In the example of the receiving unit not being the communication device, the message is sent to the receiving unit.

[0093] The device handled by a user could be, for example, an endpoint device such as a tablet, a smartphone, an Internet of Things (loT) device, or a smart controller. For example, the user could be a remote pilot controlling the UAV. In one non-restrictive example, the remote pilot initiates step 380 by receiving the message with the device operated by the user and initiating an action himself by steering the UAV via the receiving unit being the device. By receiving a message via the receiving unit, the remote pilot is easily able to monitor compliance with that the 1 :1 rule. The controller device may be, for example, a device that is not actuated by a user. In one embodiment, the controller device is capable of initiating step 380 without human intervention. For example, the control device is any device that could control the UAV, such as an endpoint device with processing circuitry, such as a tablet, smartphone, Internet of Things (loT) device, or smart controller. This enables automatic safety monitoring of the UAV. The air traffic management facility could be, for example, the U-SPACE facility in Europe, the Unmanned Traffic Management (UTM) facility in the United States, or another air traffic control facility. The facilities could for example monitor compliance with that the lateral distance is shorter than the determined value times the altitude of the UAV as a security requirement for the associated airspace.

[0094] In an embodiment issuing a message to the receiving unit comprises signaling over a short-range radio network or over a 3rdGeneration Partnership Project (3GPP) wireless network. The short-range network could be, for example, Bluetooth or Wireless Fidelity (Wi-Fi). Signaling over a wireless network may include, for example, the use of the control plane within the 3GPP wireless network or the user plane within the 3GPP wireless network. Using 3GPP standardization enables a broader ecosystem than using short-range networks because, for example, UAV power consumption can be limited by transmitting over the control plane. The use of 3GPP standardization also enables the inclusion of security capabilities embedded in 3GPP. This allows additional security measures, such as IPsec security, to be avoided.

[0095] The optional step 380 comprises preventing the UAV to move further in the direction of the detected object if the lateral distance is less or equal to the determined value times the altitude of the UAV. In other words, in case of non-compliance with the 1 :1 rule further movement towards the detected object is avoided. Preventing the UAV to move further in the direction of the detected object may be an action initiated by the remote unit. The action could be a for example, maneuver to create a greater lateral distance between the UAV and the detected object or to stop to hover at the latest when the lateral distance does not break the 1 :1 rule.

[0096] Figure 5 is a block diagram illustrating embodiments of the communication device where embodiments presented herein can be applied. The communication device being or associated with the UAV 110, wherein the UAV comprises a camera arrangement 120, comprising at least one camera 130 for creating at least one image or at least one video, wherein the communication device comprises a processing circuitry 550. In case of the communication device being the UAV, the steps described in figure 3, step 310 to 380, are performed by the UAV. If the communication is associated with the UAV, then the steps described in figure 3, step 310 to step 380, is in an embodiment performed via a remote control. The remote control may enable reduced power consumption of the UAV. The communication device associated with the UAV is in an embodiment a network device. The network device could be a network function host like a host for an Uncrewed Aerial System Network function (UAS NF), an application server host connected to the UAS NF and / or a UAS Service Supplier (USS). The network function could be run on a server host which can be accessed over the Internet, for example, in a cloud.

[0097] The communication device associated with the UAV is alternatively an end user device. The end user device could be a smartphone, a tablet, an Internet of Things (loT) device or a controlling unit, equipped with a UAV Controller (UAVC). Examples of an loT device are a wearable device or a device related to extended reality, such as a smart watch, an eyeglass, an activity tracker, or a VR headset; or a transportation vehicle, such as a robot, an aircraft, a boat, a lorry, a bus, and a car.

[0098] The UAV is an aircraft without any human pilot, crew, or passengers on board. The UAV could be for example a multi-rotor, fixed-wing, single rotor or fixed-wing hybrid vertical take-off and landing (VTOL) UAV type.

[0099] In an embodiment, the camera arrangement 120 comprises a gimbal 540 for rotation of the camera arrangement. The gimbal is a pivoted mount that allows the camera arrangement to rotate around an axis. The gimbal may have two axes or three axes, preferably three axes. The gimbal enables precise alignment of the camera arrangement and stabilizes a picture / video when shooting an image or video.

[0100] The at least one camera may be a monocular camera. Advantageously the camera may have a high resolution of at least 720 pixels, i.e. 1280 horizontal pixels and 720 vertical pixels, for a good performance. The camera arrangement 550 could comprise a first visual camera, and either a thermal camera or a second visual camera. The visual camera is for example a red green blue, RGB, camera. The present disclosure of the invention is not limited by incorporating multiple cameras into the camera arrangement. In any case, the invention is applicable using one camera. The usage of multiple cameras may increase the accuracy.

[0101] A focal length of the camera arrangement can be, for example, extracted from a focal length field of photo metadata or calculated using an image width of the image or video and a sensor width of the camera arrangement The focal length of the camera arrangement in case of two or more cameras comprised in the camera arrangement, is composed of each respective focal length of each camera. An example for suitable photo metadata is the Exchangeable Image File Format, EXIF, metadata using, for example, the EXIF tool (ExifTool, retrieved on 27 July 2023). The photo metadata is the information stored within an image describing, for example, the camera settings used or the shoot location. The EXIF specifies, for example, formats for images or ancillary tags used by digital cameras. The EXIF metadata is the file format, which stores the image metadata. The focal length of the camera arrangement any be extracted by searching a photo of the UAV with the camera arrangement mounted on in the EXIF metadata and can be extracted out of the “Calibrated Focal length” field. Another approach to obtain the focal length is to calculate the focal length in pixel using the image width of the image or video and the sensor width of the camera arrangement. The focal length in meter and the sensor width in meter is for example given by the manufacturer of the used camera arrangement. The image width could be taken from the image or video taken by the camera arrangement. The focal length in pixel is given by: f [pixel]= (Eq. 11 ) wherein f denotes for the focal length, d| [pixel] the image width in pixel and dg [m] the sensor width in meter.

[0102] Figure 6 illustrates a more detailed schematic diagram illustrating an example of an environment in which embodiments presented herein can be applied. It relates to a communication device 100 being or associated with an UAV 110. The communication device may for example a network device 602 or an end user device 604. The detected object 140 can be for example classified in step 330 as a person 642, a building 644, a vehicle 646, or any other item 648 in a surrounding environment the UAV is located such as for example a tree, a mountain, or any item with a physical height. A receiving unit 150 to which the message in step 370 is issued to, may be comprised in a device handled by a user 160 or a controller device 654 or an air traffic measurement facility 656 if the communication device 100 is being the UAV. In another example the receiving unit 150 comprises a device handled by a user 160 or a controller device 656 or an air traffic measurement facility 648 or the UAV 110 in case the communication device 100 is associated with the UAV 110.

[0103] Figure 7 illustrates a block diagram illustrating embodiments of a communication device 100 in further detail. The communication device 100 is configured to enable compliance with the condition that the lateral distance d|_| is larger to the determined value times the altitude of the UAV. In practice, the steps 310 to 380 of the method 300 performed by the communication device 100 are performed by a processing circuitry550, embodied in one or more microprocessors arranged to execute a computer program 710 that is downloaded to a computer program product 770, here in the form of a suitable computer readable storage medium 730 associated with the microprocessor, such as random access memory (RAM), read-only memory (ROM), or a non-volatile computer readable storage medium, such as flash memory or a hard disk drive, or any combination thereof. The computer program 820 comprises computer-executable instructions is stored or downloaded to the memory 730 and are executable by the processing circuitry 550. Alternatively, the computer program 710 may be transferred to the storage medium 730 using a suitable computer program product, such as a memory stick or in a memory of a device. Thus, the computer program 710 may be stored in any suitable manner in the computer program product. The processing circuity 550 is arranged to cause the communication device 100 to carry out the steps 310 to 380 of method 300 in accordance with any of the of the described embodiments for steps 310 to 380. The processing circuitry is in one embodiment a general purpose processor, but may alternatively be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), etc. An I / O interface 740 is provided for communicating with external and / or internal entities using wired communications, e.g., based on Ethernet, and / or wireless communications, e.g., Wi-Fi, and / or a cellular network corresponding to one or a combination of 5G cellular networks, LTE, LTE-advanced, UMTS, or any other current or future wireless network, such as a future 3GPP 6G network, as long as the principles described below are applicable. In one embodiment, the communication device 100 further comprises a camera arrangement 120 as described above with reference to Figure 5.

Claims

CLAIMS:1 . A communication device (100) being or associated with an Uncrewed Autonomous Vehicle, UAV, (110) wherein the UAV comprises a camera arrangement (120), comprising at least one camera (130) for creating at least one image or at least one video, and the communication device comprises a processing circuitry (550) and a memory (730), which comprises instructions executable by the processing circuitry (550) which, when executed by the processing circuitry, cause the communication device to be configured to : detect at least one object (140) from the image or video; calculate at least one lateral distance from the UAV to the detected object; compare the lateral distance with a determined value times a current altitude of the UAV; and issue a message to a receiving unit (150) if the lateral distance is less or equal to the determined value times the altitude of the UAV.

2. The communication device (100) according to claim 1 , wherein the camera arrangement comprises a gimbal (540) for rotation of the camera arrangement.

3. The communication device (100) according to any one of the previous claims, wherein the processing circuitry causes the communication device to be configured to: generate a bounding box around a projection of the detected object on an image plane of the image or video; and classify the detected object (140) into a class representing a person (642), a building (644), a vehicle (646), or a class representing any other item than a person, a building, and a vehicle (648).

4. The communication device (100) according to claim 3, wherein the calculation of the lateral distance from the UAV to the detected object is based on a focal length of the camera arrangement, a gimbal pitch of the camera arrangement , a height of the bounding box, a height of the detected object and a fraction of the upper part of thebounding box of the respective image plane, wherein the upper part is the part of the bounding box facing away from a ground on which the detected object is located.

5. The communication device (100) according to claim 4, wherein the calculation of the lateral distance from the UAV to the detected object is based onwherein d|_| is the lateral distance from the UAV to the detected object, Hj is the height of the bounding box, 0 is the gimbal pitch of the camera arrangement, f is the focal length of the camera arrangement, HR is the height of the detected object and x is the fraction of the upper part of the bounding box of the image plane.

6. The communication device (100) according to claim 4 or 5, wherein the focal length of the camera arrangement can be extracted from a focal length field of photo metadata of the image or the video or calculated using an image width of the image or video and a sensor width of the camera arrangement.

7. The communication device (100) according to any one of the previous claims, wherein the processing circuity causes the communication device to be configured to: select the shortest lateral distance if two or more objects are detected.

8. The communication device (100) according to claim 7, wherein the lateral distance is the shortest lateral distance.

9. The communication device (100) according to any of the previous claims, wherein the determined value is any number equal or greater than one.

10. The communication device (100) according to any of the previous claims, wherein the value is determined based on uncertainty data.11 . The communication device (100) according to any one of the previous claims, wherein the processing circuity causes the communication device to be configured to: prevent the UAV to move further in the direction of the detected object if the lateral distance is less or equal to the determined value times the altitude of the UAV.

12. The communication device (100) according to claim 11 , wherein the prevention of the UAV to move further in the direction of the detected object is initiated by the receiving unit.

13. The communication device (100) according to any one of the previous claims, wherein the communication device is the UAV (110) and the receiving unit (150) comprised in a device handled by a user (160) or a controller device (654) or an air traffic management facility (656).

14. The communication device (100) according to any one of the claims 1 to 12, wherein the communication device is external from the UAV (110), and the receiving unit (150) comprises a device handled by a user (160) or a controller device (654) or an air traffic management facility (656) or the UAV (110).

15. The communication device (100) according to any one of the previous claims, wherein the issue of a message to the receiving unit comprises signaling over a short range radio network or over a 3rdGeneration Partnership Project, 3GPP, wireless network.

16. The communication device (100) according to any one of the previous claims, wherein multispectral object detection is enabled by combining images or videos from one or more cameras in the camera arrangement.

17. The communication device (100) according to any one of the previous claims, wherein the calculation of the lateral distance from the UAV to the detected object is done if the detected object is classified to be a person.

18. The communication device (100) according to any one of the previous claims, wherein the height of the detected object is approximated by an average human height or estimated using a stereo-type camera arrangement (121 ), wherein the average human height is between 100 centimeter and 250 centimeter.

19. The communication device (100) according to claim 4 or 5, wherein the fraction x equals 2 if the bounding box is centered vertically to the image plane by adjusting the gimbal of the camera arrangement.

20. The communication device (100) according to any one of the previous claims, wherein the camera is a monocular camera.21 . The communication device (100) according to any one of the previous claims, wherein the camera arrangement comprises a first visual camera, and either a thermal camera or a second visual camera.

22. The communication device (100) according to any one of the previous claims, wherein the communication device is associated with the UAV and is a network device (602).

23. The communication device (100) according to claim 1 to 21 , wherein the communication device is associated with the UAV and is an end user device (604).

24. A method (200, 300) performed by a communication device (100) being or associated with an Uncrewed Autonomous Vehicle, UAV, (110), wherein the UAV comprises a camera arrangement (120) comprising at least one camera (130) for creating at least one image or video, the method comprising: detecting (210, 310) at least one object (140) from the image or video; calculating (220, 340) at least one lateral distance from the UAV to the detected object; comparing (230, 360) the lateral distance with a determined value times a current altitude of the UAV; and issuing (240, 370) a message to a receiving unit (150) if the lateral distance is less or equal to a determined value times the altitude of the UAV.

25. The method according to claim 24, wherein the camera arrangement comprises a gimbal (540) for rotation of the camera arrangement.26 The method according to any one of claims 24 to 25, wherein the method comprises: generating (320) a bounding box around a projection of the detected object on an image plane of the image or video; and classifying (330) the detected object (140) into a class representing a person (642), a building (644), a vehicle (646), or a class representing any other item than a person, a building, and a vehicle (648).

27. The method according to claim 26, wherein calculating (220, 340) the lateral distance from the UAV to the detected object is based on a focal length of thecamera arrangement, a gimbal pitch of the camera arrangement, a height of the bounding box, a height of the detected object and a fraction of the upper part of the bounding box of the respective image plane, wherein the upper part is the part of the bounding box facing away from a ground on which the detected object is located.

28. The method according to claim 27, wherein calculating (220, 340) the lateral distance from the UAV to the detected object is based onwherein d|_| is the lateral distance from the UAV to the detected object, Hj is the height of the bounding box, 0 is the gimbal pitch of the camera arrangement, f is the focal length of the camera arrangement, HR is the height of the detected object and x is the fraction of the upper part of the bounding box of the image plane.

29. The method according to claim 28 or 29, wherein the focal length of the camera arrangement can be extracted from a focal length field of photo metadata of the image or the video or calculated using an image width of the image or video and a sensor width of the camera arrangement.

30. The method according to any one of claims 24 to 29, wherein the method comprises: selecting (350) the shortest lateral distance if two or more objects are detected.31 . The method according to 30, wherein the lateral distance is the shortest lateral distance.

32. The method according to any one of claims 24 to 31 , wherein the determined value is any number equal or greater than one.

33. The method according to any one of claims 24 to 32, wherein the value is determined based on uncertainty data.

34. The method according to any one of claims 24 to 33, wherein the method comprises:preventing (380) the UAV to move further in the direction of the detected object if the lateral distance is less or equal to the determined value times the altitude of the UAV.

35. The method according to claim 34, wherein preventing (380) preventing the UAV to move further in the direction of the detected object is initiated by the receiving unit.

36. The method according to any one of claims 24 to 35, wherein the communication device (100) is the UAV (110) and the receiving unit (150) comprised in a device handled by a user (160) or a controller device (654) or an air traffic management facility (656).

37. The method according to any one of claims 24 to 35, wherein the communication device (100) is external from the UAV (110), and the receiving unit (150) comprises a device handled by a user (160) or a controller device (654) or an air traffic management facility (656) or the UAV (110).

38. The method according to any one of claims 24 to 37, wherein issuing (240, 370) a message to the receiving unit comprises signaling over a short-range radio network or over a 3rd Generation Partnership Project, 3GPP, wireless network.

39. The method according to any one of claims 24 to 38, wherein multispectral object detection is enabled by combining images or videos from one or more cameras in the camera arrangement.

40. The method according to any one of claims 24 to 39, wherein calculating (340, 220) the lateral distance from the UAV to the detected object is done if the detected object is classified to be a person.41 . The method according to any one of claims 24 to 40, wherein the height of the detected object is approximated by an average human height or estimated using a stereo-type camera arrangement (121 ), wherein the average human height is between 100 centimeter and 250 centimeter.

42. The method according to claim 27 or 28, wherein the fraction x equals 2 if the bounding box is centered vertically to the image plane by adjusting the gimbal pitch of the camera arrangement.

43. The method according to any one of claims 24 to 42, wherein the at least one camera is a monocular camera.

44. The method according to any one of claims 24 to 43, wherein the camera arrangement comprises a first visual camera, and either a thermal camera or a second visual camera.

45. The method according to any one of claims 24 to 44, wherein the communication device (100) associated with the UAV and is a network device (602).

46. The method according to any one of claims 24 to 45, wherein the communication device (100) associated with the UAV and is an end user device (604).

47. A computer program product (770) comprising a computer readable storage medium (730), the computer readable storage medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a communication device (100) being or associated with an Uncrewed Autonomous Vehicle, UAV, (110), is caused to perform a method as claimed in claim 24 to claim 45.

48. A computer program (710) comprising a computer readable code which, when run on a communication device (100) being or associated with an Uncrewed Autonomous Vehicle, UAV, causes the communication device (100) to perform the method according to claim 24 to claim 46.

49. A computer-readable storage medium (730) comprising a computer program (710) according to claim 47.

Images