Control method and device of flight equipment, storage medium and electronic device

Abstract

Embodiments of the present application provide a flight device control method and device, a storage medium and an electronic device, wherein the method comprises: extracting positioning data possessed by a positioning code from a positioning image, wherein the positioning image is an image of the positioning code deployed in a target scene collected by the flight device; determining relative position information between the flight device and the positioning code according to the positioning data, and determining coordinate information of the positioning code in a coordinate system of the flight device according to the relative position information, wherein the relative position information is used to indicate a relative position relationship between the flight device and the positioning code; and controlling the flight device to fly in the target scene according to the coordinate information. Through the present application, the problem that the complexity of the flight device positioning process is high is solved, and the complexity of the flight device positioning process is reduced.

Description

Technical Field

[0001] This application relates to the field of aircraft, and more specifically, to a control method, apparatus, storage medium, and electronic device for flight equipment. Background Technology

[0002] In the field of aircraft technology, typical application scenarios for flight equipment include sequentially detecting multiple points of interest or controlling aircraft to fly to a fixed point. These applications require flight equipment to be able to locate itself within the scene. Currently, the localization methods used by flight equipment in different scenarios are generally as follows: In outdoor scenes, RTK (Real-time Kinematic) technology is used for localization; while in indoor scenes where RTK technology is unavailable, flight equipment typically uses SLAM (Simultaneous Localization and Mapping) algorithms to construct a map of the current scene and marks the location requiring localization on the map, allowing the flight equipment to perform localization based on the map.

[0003] It is evident that the current positioning technology for flight equipment not only requires switching between different algorithms depending on the scenario, but also some algorithms have very high requirements for scenario conditions, and some algorithms have complex operation procedures. These factors result in the current positioning process for flight equipment being cumbersome and complex, and the positioning efficiency being very low.

[0004] There is still no effective solution to the problem of high complexity in the positioning process of control flight equipment in related technologies. Summary of the Invention

[0005] This application provides a control method, apparatus, storage medium, and electronic device for flight equipment, to at least solve the problem of high complexity in the control of flight equipment positioning process in related technologies.

[0006] According to one embodiment of this application, a control method for a flight device is provided, comprising:

[0007] The positioning data of the positioning code is extracted from the positioning image, wherein the positioning image is an image of the positioning code deployed in the target scene collected by the flight equipment;

[0008] The relative position information between the flight equipment and the positioning code is determined based on the positioning data, and the coordinate information of the positioning code in the coordinate system of the flight equipment is determined based on the relative position information, wherein the relative position information is used to indicate the relative positional relationship between the flight equipment and the positioning code;

[0009] The flight equipment is controlled to fly in the target scene based on the coordinate information.

[0010] In one exemplary embodiment, extracting the location data of the location code from the location image includes:

[0011] Identify the image coordinates of each of the multiple positioning points on the positioning code;

[0012] Convert the image coordinates into three-dimensional coordinates;

[0013] Multiple sets of image coordinates and three-dimensional coordinates with corresponding relationships are determined as the positioning data.

[0014] In an exemplary embodiment, identifying the image coordinates of each of the plurality of positioning points on the positioning code includes: identifying the reference coordinates of each of the four positioning angles of the positioning QR code on the positioning image, wherein the positioning code includes the positioning QR code, and the plurality of positioning points include the four positioning angles; converting each of the reference coordinates into normalized coordinates to obtain the image coordinates of the four positioning angles;

[0015] The step of converting the image coordinates into three-dimensional coordinates includes: calculating the three-dimensional coordinates of each of the four positioning angles based on the size of the positioning QR code and the image coordinates of the four positioning angles.

[0016] In an exemplary embodiment, determining the relative position information between the flight device and the positioning code based on the positioning data, and determining the coordinate information of the positioning code in the coordinate system of the flight device based on the relative position information, includes:

[0017] The relative position and relative attitude between the flight equipment and the positioning code are calculated based on multiple sets of corresponding image coordinates and three-dimensional coordinates included in the positioning data, wherein the relative position information includes the relative position and the relative attitude;

[0018] Based on the relative position, the relative attitude, and the device coordinates of the flight equipment in the northeast coordinate system, the coordinate information of the positioning code in the northeast coordinate system is calculated, wherein the coordinate system of the flight equipment includes the northeast coordinate system.

[0019] In an exemplary embodiment, calculating the relative position and relative attitude between the flight device and the positioning code based on multiple sets of corresponding image coordinates and three-dimensional coordinates included in the positioning data includes:

[0020] The constraint equations for relative position and relative attitude variables are constructed using the multiple sets of corresponding image coordinates and three-dimensional coordinates.

[0021] The constraint equations are used to construct the optimization objectives for the relative position variables and the relative attitude variables;

[0022] Solve for the optimization objective to obtain the relative position and the relative attitude.

[0023] In an exemplary embodiment, controlling the flight device to fly in the target scene based on the coordinate information includes:

[0024] The position coordinates of the target point are calculated based on the coordinate information and the reference position information between the target point in the target scene and the positioning code, wherein the reference position information is used to indicate the relative positional relationship between the target point and the positioning code;

[0025] Extract the translation and yaw angle components from the position coordinates;

[0026] The flight equipment is controlled to fly toward the target point according to the translation component and the yaw angle component.

[0027] In an exemplary embodiment, controlling the flight device to fly toward the target point according to the translation component and the yaw angle component includes:

[0028] When the positioning code is used to instruct the flight device to detect points of interest in the target scene, the flight device is controlled to fly towards the point of interest according to the translation component and the yaw angle component, and when the flight device flies to the region of interest where the point of interest is located, the flight device is controlled to stop flying;

[0029] When the positioning code is used to instruct the flight equipment to land, the flight equipment is controlled to fly toward the take-off and landing point according to the translation component and the yaw angle component, and land at the take-off and landing point, wherein the target point includes the take-off and landing point.

[0030] According to another embodiment of this application, a control device for a flight device is provided, comprising:

[0031] An extraction module is used to extract the positioning data of the positioning code from the positioning image, wherein the positioning image is an image of the positioning code deployed in the target scene collected by the flight equipment;

[0032] The determining module is used to determine the relative position information between the flight equipment and the positioning code based on the positioning data, and to determine the coordinate information of the positioning code in the coordinate system of the flight equipment based on the relative position information, wherein the relative position information is used to indicate the relative positional relationship between the flight equipment and the positioning code;

[0033] The control module is used to control the flight equipment to fly in the target scene based on the coordinate information.

[0034] According to yet another embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0035] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0036] This application describes a method where a positioning image is deployed in a target scene. The image of the positioning code acquired by the flight equipment serves as the positioning image, from which positioning data is extracted. Based on the extracted positioning data, relative positional information indicating the relative positional relationship between the flight equipment and the positioning code is determined. Furthermore, the coordinates of the positioning code within the flight equipment's coordinate system are determined based on this relative positional relationship, allowing for flight control. This positioning process does not place excessive demands on scene conditions and eliminates the need for complex operations such as constructing scene maps and marking locations. It only requires extracting the positioning data of the positioning code deployed in the scene. By determining the relative positional relationship and coordinate information, the flight equipment can be controlled in any scene, enabling adaptive positioning control without switching algorithms for different scenes or deploying multiple positioning algorithms on the flight equipment. Therefore, it solves the problem of high complexity in the positioning process of flight equipment control, effectively reducing the complexity of the positioning process. Attached Figure Description

[0037] Figure 1 This is a hardware structure block diagram of a terminal device for a flight equipment control method according to an embodiment of this application;

[0038] Figure 2 This is a flowchart of a control method for a flight device according to an embodiment of this application;

[0039] Figure 3 This is a schematic diagram of a drone positioning process based on six-directional QR code vision, according to an embodiment of this application. Figure 1 ;

[0040] Figure 4 This is a schematic diagram of a drone positioning process based on six-directional QR code vision, according to an embodiment of this application. Figure 2 ;

[0041] Figure 5 This is a schematic diagram of a control flow for a drone according to an embodiment of this application;

[0042] Figure 6 This is a structural block diagram of a control device for a flight equipment according to an embodiment of this application. Detailed Implementation

[0043] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0044] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0045] The methods and embodiments provided in this application can be executed in drones, aircraft, or similar terminal devices. Taking running on a drone as an example, Figure 1 This is a hardware structure block diagram of a terminal device for a flight equipment control method according to an embodiment of this application. For example... Figure 1 As shown, the drone includes a body 102, propellers 104, and may also include one or more ( Figure 1 Only one is shown in the image. A processor 106 (processor 106 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 108 for storing data are also shown. The aforementioned UAV may further include a transmission device 110 for communication functions and one or more camera devices 112. Figure 1 Only one is shown in the image. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, a drone may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0046] The memory 108 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the control method of the flight equipment in this embodiment. The processor 106 executes various functional applications and data processing by running the computer program stored in the memory 108, thereby implementing the above-described method. The memory 108 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 108 may further include memory remotely located relative to the processor 106, and these remote memories can be connected to the drone via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0047] Transmission device 110 is used to receive or send data via a network. In one example, transmission device 110 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0048] This embodiment provides a control method for flight equipment. Figure 2 This is a flowchart of a control method for a flight device according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0049] Step S202: Extract the positioning data of the positioning code from the positioning image, wherein the positioning image is an image of the positioning code deployed in the target scene collected by the flight equipment;

[0050] Step S204: Determine the relative position information between the flight equipment and the positioning code based on the positioning data, and determine the coordinate information of the positioning code in the coordinate system of the flight equipment based on the relative position information, wherein the relative position information is used to indicate the relative positional relationship between the flight equipment and the positioning code;

[0051] Step S206: Control the flight device to fly in the target scene according to the coordinate information.

[0052] Through the above steps, the positioning image is deployed in the target scene. The image of the positioning code acquired by the flight equipment is used as the positioning image, from which positioning data is extracted. Then, based on the extracted positioning data, relative position information indicating the relative positional relationship between the flight equipment and the positioning code is determined. Furthermore, based on the relative positional relationship between the flight equipment and the positioning code, the coordinate information of the positioning code in the coordinate system of the flight equipment is determined, thereby controlling the flight of the flight equipment based on this coordinate information. This positioning process does not have high requirements for scene conditions and does not require complex operations such as constructing scene maps and marking positions on maps. It only needs to extract the positioning data of the positioning code deployed in the scene. By determining the relative positional relationship and coordinate information, the flight process of the flight equipment can be controlled in any scene. This allows for adaptable positioning control of the flight equipment in any scene, without switching different algorithms based on scene changes or deploying multiple positioning algorithms on the flight equipment. Therefore, it can solve the problem of high complexity in the positioning process of controlling flight equipment, achieving the effect of reducing the complexity of the positioning process.

[0053] Optionally, in this embodiment, the control process of the flight equipment described above can be applied to, but is not limited to, any device with flight equipment control functions. For example: the flight equipment, the remote controller of the flight equipment, the terminal connected to the flight equipment (mobile phone, tablet computer, personal computer), etc.

[0054] In the solution provided in step S202 above, the positioning image may include, but is not limited to, any type of image. For example, frames in a video, photographs taken, etc.

[0055] Optionally, in this embodiment, the location code may include, but is not limited to, any type of encoding for location. For example, a reference marker (e.g., Apriltag), a location QR code, etc.

[0056] Optionally, in this embodiment, the flight device can, but is not limited to, detect the presence of the positioning code in the target scene through a deployed six-way vision system, thereby avoiding frequent rotation of the flight device and enabling it to discover the positioning code in the target scene more quickly and conveniently, thus improving the positioning efficiency of the flight device.

[0057] Optionally, in this embodiment, the target scenario may include, but is not limited to, any indoor or outdoor scenario that allows the deployment of location codes. For example: rooms, stadiums, playgrounds, parking lots, offices, etc.

[0058] Optionally, in this embodiment, the aforementioned flight equipment may include, but is not limited to, any type of aircraft that is allowed to reach a location point based on positioning data. For example, drones, conventional aircraft, etc.

[0059] Optionally, in this embodiment, the positioning data may include, but is not limited to, any data used to locate the location information of the positioning code. For example, this data may be encoded and edited into the pattern of the positioning code, or the positioning data may be parsed from the features of the pattern of the positioning code through processes such as extraction and calculation.

[0060] In one exemplary embodiment, the positioning data of the positioning code can be extracted from the positioning image by, but is not limited to, identifying the image coordinates of each positioning point among a plurality of positioning points on the positioning code; converting the image coordinates into three-dimensional coordinates; and determining a plurality of sets of image coordinates and three-dimensional coordinates with corresponding relationships as the positioning data.

[0061] Optionally, in this embodiment, the positioning code may, but is not limited to, have multiple positioning points (e.g., corner points, edge points, center points, etc.). The image coordinates of each positioning point may, but are not limited to, refer to the coordinates of the positioning point in space projected onto the image captured by the camera of the flight device. The three-dimensional coordinates of the positioning point may, but are not limited to, refer to the actual coordinates of the positioning point in space.

[0062] Optionally, in this embodiment, the multiple positioning points may be, but are not limited to, pre-set on the positioning code, or may be, but are not limited to, points with better positioning performance selected from multiple points that can be identified on the positioning code based on the positioning effect.

[0063] In one exemplary embodiment, the image coordinates of each of the plurality of positioning points on the positioning code can be identified, but is not limited to, by: identifying the reference coordinates of each of the four positioning angles of the positioning QR code on the positioning image, wherein the positioning code includes the positioning QR code, and the plurality of positioning points include the four positioning angles; converting each of the reference coordinates into normalized coordinates to obtain the image coordinates of the four positioning angles;

[0064] In addition, image coordinates can be converted into three-dimensional coordinates by, but not limited to, calculating the three-dimensional coordinates of each of the four positioning angles based on the size of the positioning QR code and the image coordinates of the four positioning angles.

[0065] Optionally, in this embodiment, the multiple positioning points may be, but are not limited to, the four corner points of the positioning code (i.e., the aforementioned positioning angles), thereby obtaining four sets of corresponding image coordinates and three-dimensional coordinates, which can solve the relative positional relationship between the positioning code and the flight equipment.

[0066] Optionally, in this embodiment, the reference coordinates of the identified positioning angles on the image are normalized to project the three-dimensional points onto the same line, thereby obtaining the image coordinates of each positioning angle.

[0067] Optionally, in this embodiment, the size of the positioning QR code may be, but is not limited to, pre-obtained, or may be obtained through measurement.

[0068] In one optional implementation, Apriltag is used as the location QR code, with a size of 50cm x 50cm. After the flight equipment identifies the location QR code using a six-directional vision system, it can obtain the coordinates u of the four corner points of the location QR code on the image. pi (i.e., the aforementioned reference coordinates), which are then transformed into normalized coordinates v. pi Thus, the above image coordinates are obtained. Since the size of the positioning QR code is known, the three-dimensional coordinates p of the four corner points can be obtained. pi .

[0069] In the solution provided in step S204 above, the relative position information is used to indicate the relative positional relationship between the flight equipment and the positioning code. For example, but not limited to, information such as the relative position and relative attitude between the flight equipment and the positioning code can be used as the relative position information.

[0070] Optionally, in this embodiment, the relative positional relationship between the flight equipment and the positioning code is first determined based on the positioning data extracted from the positioning image. Then, the positioning code can be placed in the coordinate system of the flight equipment using this relative positional relationship to determine the coordinate information of the positioning code, thereby realizing the positioning of the positioning code in the target scene.

[0071] Optionally, in this embodiment, the coordinate system of the flight equipment may include, but is not limited to, the North East Down (NED) coordinate system, the Earth's central coordinate system, WGS-84 (The WGS-84 coordinate system, 1984 World Geodetic Coordinate System), etc.

[0072] In one exemplary embodiment, the relative position information between the flight device and the positioning code can be determined based on positioning data in the following manner, but not limited to: calculating the relative position and relative attitude between the flight device and the positioning code based on multiple sets of corresponding image coordinates and three-dimensional coordinates included in the positioning data, wherein the relative position information includes the relative position and the relative attitude; calculating the coordinate information of the positioning code in the northeast-east coordinate system based on the relative position, the relative attitude, and the device coordinates of the flight device in the northeast-east coordinate system, wherein the coordinate system of the flight device includes the northeast-east coordinate system.

[0073] Optionally, in this embodiment, the relative position and relative attitude mentioned above may be obtained by estimating the camera pose on the flight equipment. For example, the relative position and relative attitude between the flight equipment and the positioning code may be calculated using the PNP (Pespective-N-Point) algorithm, but not limited to this.

[0074] Optionally, in this embodiment, the flight device can, but is not limited to, locate the position of the positioning code in a northeast-northeast coordinate system. The device coordinates in the northeast-northeast coordinate system can, but are not limited to, be obtained in advance.

[0075] In one exemplary embodiment, the relative position and relative attitude between the flight device and the positioning code can be calculated, but is not limited to, based on multiple sets of corresponding image coordinates and three-dimensional coordinates included in the positioning data, in the following manner: using the multiple sets of corresponding image coordinates and three-dimensional coordinates to construct constraint equations for relative position variables and relative attitude variables; using the constraint equations to construct optimization objectives for the relative position variables and the relative attitude variables; solving the optimization objectives to obtain the relative position and the relative attitude.

[0076] Optionally, in this embodiment, the optimization objective can be solved by, but is not limited to, nonlinear optimization or other methods.

[0077] Optionally, in this embodiment, the constraint equations may be constructed by projecting three-dimensional points onto the same line based on the normalization of reference coordinates identified on the positioning image.

[0078] In the above optional implementation, due to the normalized coordinates v of the point pi 3D point p pi Since they are on the same line, we can obtain the following equation: Among them, R cr t is a relative attitude variable. cr It is a relative position variable.

[0079] Simplify the above equation and let Then we have:

[0080]

[0081] Therefore, we get:

[0082]

[0083] Where A [i：.] To retrieve the i-th row of matrix A.

[0084] Therefore, a line can provide one constraint, and a point can provide two constraints. The estimator R... cr, t cr With 6 degrees of freedom, only 6 constraint equations are needed to solve it.

[0085] The specific solution process is as follows:

[0086] The following optimization objectives are defined:

[0087]

[0088] in:

[0089]

[0090] First find

[0091]

[0092] So:

[0093] Seeking again

[0094]

[0095] Where: A = IW(W) T W) -1 W T .

[0096] The above optimization objective is solved using the following nonlinear optimization process:

[0097] Step 1, calculate the error:

[0098] e = AX

[0099] R cr Addition is defined as:

[0100]

[0101] Step 2, calculate the Jacobi:

[0102]

[0103] Step 3, GN Iterative Optimization Process:

[0104] J T Jδξ=-Je

[0105] Step 4: Solve the above linear equations to obtain δξ, and update R. cr :

[0106] R cr =exp(δξ)R cr

[0107] Repeat steps one through four above until δξ is less than a certain threshold to obtain the relative position R. cr Then the relative attitude t is obtained. cr :

[0108] t cr =-(W T W) -1 W T X.

[0109] Obtain the relative pose T of the aircraft from the positioning code drone_tag (i.e. R) cr and t cr After that, the aircraft's position T in the NED coordinate system is known. ned_drone This allows us to obtain the location of the positioning code in the NED coordinate system (i.e., the coordinate information mentioned above):

[0110] T ned_tag =T ned_drone T deone-tag .

[0111] In the solution provided in step S206 above, the flight equipment can, but is not limited to, perform tasks such as landing, reconnaissance, patrol, and monitoring in the target scenario. Different tasks can, but are not limited to, use different methods to control the flight process of the flight equipment.

[0112] In one exemplary embodiment, the flight device can be controlled to fly in the target scene based on coordinate information in the following manner, but not limited to: calculating the position coordinates of the target point based on the coordinate information and reference position information between the target point in the target scene and the positioning code, wherein the reference position information is used to indicate the relative positional relationship between the target point and the positioning code; extracting translational and yaw angle components from the position coordinates; and controlling the flight device to fly towards the target point according to the translational and yaw angle components.

[0113] Optionally, in this embodiment, the target point in the target scene may be, but is not limited to, a pre-set point that the flight equipment needs to reach. The reference position information between the target point in the target scene and the positioning code is used to indicate the relative positional relationship between the target point and the positioning code, and this relative positional relationship can be obtained when setting the target point.

[0114] Optionally, in this embodiment, the relative position between the target point and the positioning code is: T tag_target The position of the target point in the NED coordinate system can be obtained in advance and ultimately used as the position coordinates of the target point.

[0115] T ned_target =T ned_tag T tag_target

[0116] Optionally, in this embodiment, from T ned_target Extract the translation component t ned_target And the yaw component, thereby controlling the flight equipment to fly to the corresponding position.

[0117] In one exemplary embodiment, the flight equipment may be controlled to fly toward the target point in different ways according to different situations, but not limited to:

[0118] Method 1: When the positioning code is used to instruct the flight device to detect points of interest in the target scene, the flight device is controlled to fly towards the point of interest according to the translation component and the yaw angle component, and when the flight device flies to the region of interest where the point of interest is located, the flight device is controlled to stop flying.

[0119] Method 2: When the positioning code is used to instruct the flight equipment to land, the flight equipment is controlled to fly towards the take-off and landing point according to the translation component and the yaw angle component, and land at the take-off and landing point, wherein the target point includes the take-off and landing point.

[0120] Optionally, in this embodiment, if the target point is a point of interest that the flight equipment needs to detect, the flight equipment can be controlled to fly towards the point of interest according to the translation component and the yaw angle component. When the error between the position of the flight equipment and the position of the point of interest is less than a certain range (position error < 15cm, angle error < 5deg), it is considered that the flight equipment has flown into the area of ​​interest where the point of interest is located, that is, the aircraft has reached the target point. At this time, the flight equipment can be controlled to stop flying.

[0121] Optionally, in this embodiment, if the target point is the take-off and landing point (i.e., the home point) where the flight equipment needs to land, then the precise landing of the flight equipment can be achieved by controlling the flight equipment. Controlling the flight equipment to fly to the corresponding position of the take-off and landing point to perform the landing operation can achieve a high-precision landing of the flight equipment, with a position error of <10cm and an angle error of <5 degrees.

[0122] In this embodiment, a drone positioning process based on six-directional QR code vision is also provided. It does not require pre-construction of a map. It obtains the relative position and relative attitude between the positioning QR code and the drone simply by positioning the QR code, thereby achieving high-precision positioning of points of interest or take-off and landing points in the scene. Figure 3 This is a schematic diagram of a drone positioning process based on six-directional QR code vision, according to an embodiment of this application. Figure 1 ,like Figure 3 As shown, the process may include, but is not limited to, the following steps:

[0123] Step 31: Deploy location QR codes at multiple points of interest in the indoor scene. Deploy location QR code A at point of interest 1, location QR code B at point of interest 2, and location QR code C at point of interest 3.

[0124] Step 32: The drone uses a six-way binocular vision system to identify the location QR code in the indoor scene, for example, it first identifies the location QR code B deployed at point of interest 2.

[0125] Step 33: Calculate the relative position and relative attitude between the positioning QR code B and the UAV, and determine the coordinate information of the positioning QR code B in the UAV's NED coordinate system.

[0126] Step 34: Determine the position coordinates of point of interest 2, extract the translation component and yaw angle component from them, and control the UAV to fly accurately to the point of interest according to the translation component and yaw angle component.

[0127] Step 35: Repeat steps 32 to 34 above to control the drone to detect other points of interest 3 and point of interest 1.

[0128] The aforementioned positioning process significantly improves the accuracy of point-of-interest (POI) localization in indoor scenarios, enabling drones to accurately detect POIs indoors. Furthermore, it fully utilizes the drone's six-directional binocular vision system, increasing the success rate of QR code recognition and avoiding recognition failures due to a single-path vision system's inability to observe the QR code. In addition, while achieving accurate positioning, it also reduces the complexity of the algorithm and lowers the computational performance requirements.

[0129] The aforementioned positioning process can also enable precise landing control of drones both indoors and outdoors. Taking precise landing control of drones in outdoor scenarios as an example... Figure 4 This is a schematic diagram of a drone positioning process based on six-directional QR code vision, according to an embodiment of this application. Figure 2 ,like Figure 4As shown, the UAV takes off from the take-off and landing point and flies towards the cruise point (equivalent to the aforementioned point of interest). Upon acquiring the positioning image of the cruise point, it extracts the positioning data of positioning QR code 1 from the positioning image. Based on this positioning data, it determines the relative position information between the UAV and positioning QR code 1, and then determines the coordinates of positioning QR code 1 in the UAV's coordinate system. The UAV is then controlled to fly towards the cruise point according to these coordinates. When the UAV reaches the region of interest where the cruise point is located, it stops flying and performs a cruise mission for that cruise point. After the cruise mission is completed, the UAV flies from the cruise point towards the take-off and landing point. Upon acquiring the positioning image of the take-off and landing point, it extracts the positioning data of positioning QR code 2 from the positioning image. Based on this positioning data, it determines the relative position information between the UAV and positioning QR code 2, and then determines the coordinates of positioning QR code 2 in the UAV's coordinate system. The UAV is then controlled to fly towards the take-off and landing point according to these coordinates, and finally, based on the relative position information between the UAV and positioning QR code 2, it lands at the take-off and landing point.

[0130] In an optional embodiment, taking a drone as an example of a flight device, a control process for the drone based on machine vision is also provided. Figure 5 This is a schematic diagram of the control flow of a drone according to an embodiment of this application, such as... Figure 5 As shown, the process may include, but is not limited to, the following steps:

[0131] Step S502: Identify the reference coordinates of each of the four positioning angles of the positioning QR code in the scene on the positioning image.

[0132] Step S504: Convert each reference coordinate into normalized coordinates to obtain the image coordinates of the four positioning angles.

[0133] Step S506: Calculate the three-dimensional coordinates of each positioning angle based on the size of the positioning QR code and the image coordinates of the four positioning angles.

[0134] Step S508: Determine the corresponding sets of image coordinates and three-dimensional coordinates as positioning data.

[0135] Step S510: Use multiple sets of corresponding image coordinates and 3D coordinates to construct constraint equations for relative position variables and relative attitude variables.

[0136] Step S512: Use constraint equations to construct optimization objectives for relative position and relative attitude variables.

[0137] Step S514: Solve for the optimization objective to obtain the relative position and relative attitude.

[0138] Step S516: Calculate the coordinate information of the positioning code in the northeast coordinate system based on the relative position, relative attitude, and the equipment coordinates of the flight equipment in the northeast coordinate system.

[0139] Step S518: Calculate the position coordinates of the target point based on the coordinate information and the reference position information between the target point and the positioning code in the target scene.

[0140] Step S520: Extract the translation component and yaw angle component from the position coordinates.

[0141] Step S522: When the positioning code is used to instruct the flight equipment to detect points of interest in the target scene, control the flight equipment to fly towards the points of interest according to the translation component and the yaw angle component, and control the flight equipment to stop flying when the flight equipment flies to the area of ​​interest where the points of interest are located.

[0142] Step S524: When the positioning code is used to instruct the flight equipment to land, control the flight equipment to fly towards the landing point according to the translation component and the yaw angle component, and land at the landing point.

[0143] Through the above process, drones can accurately perform operations such as patrol reconnaissance, take-off, and landing through a simple positioning process. While ensuring operational accuracy, it can reduce the complexity of the drone positioning process, thereby improving the drone control efficiency.

[0144] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0145] This embodiment also provides a control device for a flight device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0146] Figure 6 This is a structural block diagram of a control device for a flight equipment according to an embodiment of this application, such as... Figure 6As shown, the device includes:

[0147] Extraction module 62 is used to extract the positioning data of the positioning code from the positioning image, wherein the positioning image is an image of the positioning code deployed in the target scene collected by the flight equipment;

[0148] The determining module 64 is used to determine the relative position information between the flight equipment and the positioning code based on the positioning data, and to determine the coordinate information of the positioning code in the coordinate system of the flight equipment based on the relative position information, wherein the relative position information is used to indicate the relative positional relationship between the flight equipment and the positioning code;

[0149] The control module 66 is used to control the flight device to fly in the target scene according to the coordinate information.

[0150] In one exemplary embodiment, the extraction module includes:

[0151] The identification unit is used to identify the image coordinates of each of the multiple positioning points on the positioning code;

[0152] A conversion unit is used to convert the image coordinates into three-dimensional coordinates;

[0153] The determining unit is used to determine multiple sets of image coordinates and three-dimensional coordinates with corresponding relationships as the positioning data.

[0154] In an exemplary embodiment, the identification unit is configured to: identify the reference coordinates of each of the four positioning angles of the positioning QR code on the positioning image, wherein the positioning code includes the positioning QR code, and the plurality of positioning points include the four positioning angles; and convert each of the reference coordinates into normalized coordinates to obtain the image coordinates of the four positioning angles.

[0155] The conversion unit is configured to: calculate the three-dimensional coordinates of each of the positioning angles based on the size of the positioning QR code and the image coordinates of the four positioning angles.

[0156] In one exemplary embodiment, the determining module includes:

[0157] The first calculation unit is used to calculate the relative position and relative attitude between the flight equipment and the positioning code based on multiple sets of corresponding image coordinates and three-dimensional coordinates included in the positioning data, wherein the relative position information includes the relative position and the relative attitude;

[0158] The second calculation unit is used to calculate the coordinate information of the positioning code in the northeast coordinate system based on the relative position, the relative attitude and the equipment coordinates of the flight equipment in the northeast coordinate system, wherein the coordinate system of the flight equipment includes the northeast coordinate system.

[0159] In one exemplary embodiment, the determining module includes:

[0160] The first construction unit is used to construct constraint equations for relative position variables and relative pose variables using the multiple sets of corresponding image coordinates and three-dimensional coordinates;

[0161] The second construction unit is used to construct the optimization objectives of the relative position variables and the relative attitude variables using the constraint equations;

[0162] The solution unit is used to solve the optimization objective to obtain the relative position and the relative attitude.

[0163] In one exemplary embodiment, the control module includes:

[0164] The third calculation unit is used to calculate the position coordinates of the target point based on the coordinate information and the reference position information between the target point in the target scene and the positioning code, wherein the reference position information is used to indicate the relative positional relationship between the target point and the positioning code;

[0165] Extraction unit, used to extract translation component and yaw angle component from the position coordinates;

[0166] A control unit is used to control the flight equipment to fly toward the target point according to the translation component and the yaw angle component.

[0167] In one exemplary embodiment, the control unit is configured to:

[0168] When the positioning code is used to instruct the flight device to detect points of interest in the target scene, the flight device is controlled to fly towards the point of interest according to the translation component and the yaw angle component, and when the flight device flies to the region of interest where the point of interest is located, the flight device is controlled to stop flying;

[0169] When the positioning code is used to instruct the flight equipment to land, the flight equipment is controlled to fly toward the take-off and landing point according to the translation component and the yaw angle component, and land at the take-off and landing point, wherein the target point includes the take-off and landing point.

[0170] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0171] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0172] In this embodiment, the computer-readable storage medium described above may be configured to store a computer program for performing the following steps:

[0173] S1, extract the positioning data of the positioning code from the positioning image, wherein the positioning image is an image of the positioning code deployed in the target scene collected by the flight equipment;

[0174] S2, determine the relative position information between the flight equipment and the positioning code based on the positioning data, and determine the coordinate information of the positioning code in the coordinate system of the flight equipment based on the relative position information, wherein the relative position information is used to indicate the relative positional relationship between the flight equipment and the positioning code;

[0175] S3, control the flight equipment to fly in the target scene according to the coordinate information.

[0176] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0177] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0178] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0179] In one exemplary embodiment, the processor described above may be configured to perform the following steps via a computer program:

[0180] S1, extract the positioning data of the positioning code from the positioning image, wherein the positioning image is an image of the positioning code deployed in the target scene collected by the flight equipment;

[0181] S2, determine the relative position information between the flight equipment and the positioning code based on the positioning data, and determine the coordinate information of the positioning code in the coordinate system of the flight equipment based on the relative position information, wherein the relative position information is used to indicate the relative positional relationship between the flight equipment and the positioning code;

[0182] S3, control the flight equipment to fly in the target scene according to the coordinate information.

[0183] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0184] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0185] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A control method for a flight device, characterized in that, include: The positioning data of the positioning code is extracted from the positioning image, wherein the positioning image is an image of the positioning code deployed in the target scene collected by the flight equipment; The relative position information between the flight equipment and the positioning code is determined based on the positioning data, and the coordinate information of the positioning code in the coordinate system of the flight equipment is determined based on the relative position information, wherein the relative position information is used to indicate the relative positional relationship between the flight equipment and the positioning code; The flight equipment is controlled to fly in the target scene based on the coordinate information; The step of extracting the positioning data of the positioning code from the positioning image includes: identifying the image coordinates of each positioning point among multiple positioning points on the positioning code; converting the image coordinates into three-dimensional coordinates; and determining multiple sets of image coordinates and three-dimensional coordinates with corresponding relationships as the positioning data. The step of identifying the image coordinates of each of the multiple positioning points on the positioning code includes: identifying the reference coordinates of each of the four positioning angles of the positioning QR code on the positioning image, wherein the positioning code includes the positioning QR code, and the multiple positioning points include the four positioning angles; converting each reference coordinate into normalized coordinates to obtain the image coordinates of the four positioning angles; and converting the image coordinates into three-dimensional coordinates includes: calculating the three-dimensional coordinates of each positioning angle based on the size of the positioning QR code and the image coordinates of the four positioning angles. The step of determining the relative position information between the flight device and the positioning code based on the positioning data, and determining the coordinate information of the positioning code in the coordinate system of the flight device based on the relative position information, includes: calculating the relative position and relative attitude between the flight device and the positioning code based on multiple sets of corresponding image coordinates and three-dimensional coordinates included in the positioning data, wherein the relative position information includes the relative position and the relative attitude; and calculating the coordinate information of the positioning code in the northeast-northeast coordinate system based on the relative position, the relative attitude, and the device coordinates of the flight device in the northeast-northeast coordinate system, wherein the coordinate system of the flight device includes the northeast-northeast coordinate system. The step of calculating the relative position and relative attitude between the flight device and the positioning code based on multiple sets of corresponding image coordinates and three-dimensional coordinates included in the positioning data includes: constructing constraint equations for relative position variables and relative attitude variables using the multiple sets of corresponding image coordinates and three-dimensional coordinates; constructing optimization objectives for the relative position variables and the relative attitude variables using the constraint equations; and solving the optimization objectives to obtain the relative position and the relative attitude. The step of constructing constraint equations for relative position and relative attitude variables using the multiple sets of corresponding image coordinates and three-dimensional coordinates includes: constructing four constraint equations for relative position and relative attitude variables based on the image coordinates and three-dimensional coordinates of each of the four positioning angles of the positioning QR code, wherein the constraint equations include four constraint equations, each of which is: , The image coordinates of the i-th positioning angle. Let i be the three-dimensional coordinate of the i-th positioning angle, where i takes values ​​from 1 to 4. For the relative attitude variable, The relative position variable is denoted as .

2. The method according to claim 1, characterized in that, The step of controlling the flight device to fly in the target scene based on the coordinate information includes: The position coordinates of the target point are calculated based on the coordinate information and the reference position information between the target point in the target scene and the positioning code, wherein the reference position information is used to indicate the relative positional relationship between the target point and the positioning code; Extract the translation and yaw angle components from the position coordinates; The flight equipment is controlled to fly toward the target point according to the translation component and the yaw angle component.

3. The method according to claim 2, characterized in that, Controlling the flight equipment to fly towards the target point according to the translation component and the yaw angle component includes: When the positioning code is used to instruct the flight device to detect points of interest in the target scene, the flight device is controlled to fly towards the point of interest according to the translation component and the yaw angle component, and when the flight device flies to the region of interest where the point of interest is located, the flight device is controlled to stop flying; When the positioning code is used to instruct the flight equipment to land, the flight equipment is controlled to fly toward the take-off and landing point according to the translation component and the yaw angle component, and land at the take-off and landing point, wherein the target point includes the take-off and landing point.

4. A control device for flight equipment, characterized in that, include: An extraction module is used to extract the positioning data of the positioning code from the positioning image, wherein the positioning image is an image of the positioning code deployed in the target scene collected by the flight equipment; The determining module is used to determine the relative position information between the flight equipment and the positioning code based on the positioning data, and to determine the coordinate information of the positioning code in the coordinate system of the flight equipment based on the relative position information, wherein the relative position information is used to indicate the relative positional relationship between the flight equipment and the positioning code; The control module is used to control the flight equipment to fly in the target scene according to the coordinate information; The extraction module includes: an identification unit for identifying the image coordinates of each of the multiple positioning points on the positioning code; a conversion unit for converting the image coordinates into three-dimensional coordinates; and a determination unit for determining multiple sets of image coordinates and three-dimensional coordinates with corresponding relationships as the positioning data. The identification unit is configured to: identify the reference coordinates of each of the four positioning angles of the positioning QR code on the positioning image, wherein the positioning code includes the positioning QR code, and the plurality of positioning points include the four positioning angles; convert each reference coordinate into normalized coordinates to obtain the image coordinates of the four positioning angles; the conversion unit is configured to: calculate the three-dimensional coordinates of each positioning angle based on the size of the positioning QR code and the image coordinates of the four positioning angles; The determining module includes: a first calculation unit, configured to calculate the relative position and relative attitude between the flight device and the positioning code based on multiple sets of corresponding image coordinates and three-dimensional coordinates included in the positioning data, wherein the relative position information includes the relative position and the relative attitude; and a second calculation unit, configured to calculate the coordinate information of the positioning code in the northeast coordinate system based on the relative position, the relative attitude, and the device coordinates of the flight device in the northeast coordinate system, wherein the coordinate system of the flight device includes the northeast coordinate system. The determining module includes: a first construction unit, used to construct constraint equations for relative position variables and relative pose variables using the multiple sets of corresponding image coordinates and three-dimensional coordinates; a second construction unit, used to construct optimization objectives for the relative position variables and relative pose variables using the constraint equations; and a solving unit, used to solve the optimization objectives to obtain the relative position and relative pose. The first construction unit is further configured to: construct four constraint equations for relative position variables and relative pose variables based on the image coordinates and three-dimensional coordinates of each of the four positioning angles of the positioning QR code, wherein the constraint equations include four constraint equations, each of which is: , The image coordinates of the i-th positioning angle. Let i be the three-dimensional coordinate of the i-th positioning angle, where i takes values ​​from 1 to 4. For the relative attitude variable, The relative position variable is denoted as .

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 3.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 3.

Images