Environment image acquisition method and device, vehicle-mounted unmanned aerial vehicle, monitoring system, medium and product

Abstract

The invention discloses an environment image acquisition method and device, a vehicle-mounted unmanned aerial vehicle, a monitoring system, a medium and a product. The method comprises the steps that the target pose of the vehicle-mounted unmanned aerial vehicle and the current pose of the vehicle-mounted unmanned aerial vehicle are acquired, the target pose is the expected pose of the vehicle-mounted unmanned aerial vehicle relative to the vehicle, and the current pose is the current pose of the vehicle-mounted unmanned aerial vehicle relative to the vehicle; generating a target control instruction according to the target pose of the vehicle-mounted unmanned aerial vehicle, the current pose of the vehicle-mounted unmanned aerial vehicle and the driving related data of the vehicle; and after the vehicle-mounted unmanned aerial vehicle executes the target control instruction, acquiring an environment image around the vehicle, and sending the environment image around the vehicle to the vehicle, through the technical scheme of the invention, the comprehensiveness and accuracy of the acquired environment image can be improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of unmanned aerial vehicles, and in particular, to a method and device for obtaining environmental images, an in-vehicle unmanned aerial vehicle, a monitoring system, a medium and a product. Background Art

[0002] With the continuous increase in the number of motor vehicles, the road traffic conditions have become increasingly complex, and the safety risks during vehicle driving have also increased accordingly.

[0003] Existing vehicle safety monitoring systems mainly rely on devices such as in-vehicle cameras and radars. However, in-vehicle cameras have blind spots in the viewing angle and it is difficult to comprehensively cover all areas around the vehicle body; although radars can detect the distance and approximate direction of obstacles, they cannot provide intuitive image information and it is difficult to accurately identify the type and state of objects.

[0004] Since drivers need more comprehensive and accurate information about the environment around the vehicle body to avoid collision accidents. Therefore, how to obtain more comprehensive and accurate information about the environment around the vehicle body has become an urgent problem to be solved currently. Summary of the Invention

[0005] The embodiments of the present invention provide a method and device for obtaining environmental images, an in-vehicle unmanned aerial vehicle, a monitoring system, a medium and a product, which can improve the comprehensiveness and accuracy of the obtained environmental images.

[0006] According to one aspect of the present invention, there is provided a method for obtaining environmental images, including:

[0007] Obtaining a target pose and a current pose of the in-vehicle unmanned aerial vehicle, where the target pose is the expected pose of the in-vehicle unmanned aerial vehicle relative to the vehicle, and the current pose is the current pose of the in-vehicle unmanned aerial vehicle relative to the vehicle;

[0008] Generating a target control instruction according to the target pose of the in-vehicle unmanned aerial vehicle, the current pose of the in-vehicle unmanned aerial vehicle, and the driving-related data of the vehicle;

[0009] After the in-vehicle unmanned aerial vehicle executes the target control instruction, obtaining an environmental image around the vehicle and sending the environmental image around the vehicle to the vehicle.

[0010] According to another aspect of the present invention, there is provided an apparatus for obtaining environmental images, and the apparatus for obtaining environmental images includes:

[0011] An obtaining module, configured to obtain a target pose and a current pose of the in-vehicle unmanned aerial vehicle, where the target pose is the expected pose of the in-vehicle unmanned aerial vehicle relative to the vehicle, and the current pose is the current pose of the in-vehicle unmanned aerial vehicle relative to the vehicle;

[0012] A target control instruction generation module, configured to generate a target control instruction according to the target pose of the vehicle-mounted unmanned aerial vehicle, the current pose of the vehicle-mounted unmanned aerial vehicle, and the driving-related data of the vehicle;

[0013] An environmental image sending module, configured to, after the vehicle-mounted unmanned aerial vehicle executes the target control instruction, acquire an environmental image around the vehicle and send the environmental image around the vehicle to the vehicle.

[0014] According to another aspect of the present invention, there is provided a vehicle-mounted unmanned aerial vehicle, including:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor is enabled to execute the environmental image acquisition method according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, there is provided a monitoring system, including: a vehicle-mounted unmanned aerial vehicle and a vehicle, and the vehicle-mounted unmanned aerial vehicle is capable of executing the environmental image acquisition method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for enabling a processor to implement the environmental image acquisition method according to any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, there is provided a computer program product, and when the computer program is executed by a processor, it implements the environmental image acquisition method according to any one of the embodiments of the present invention.

[0021] In an embodiment of the present invention, the target pose of the vehicle-mounted drone and the current pose of the vehicle-mounted drone are obtained, where the target pose is the desired pose of the vehicle-mounted drone relative to the vehicle, and the current pose is the current pose of the vehicle-mounted drone relative to the vehicle; according to the target pose of the vehicle-mounted drone, the current pose of the vehicle-mounted drone, and the driving-related data of the vehicle, a target control instruction is generated; after the vehicle-mounted drone executes the target control instruction, an environmental image around the vehicle is obtained, and the environmental image around the vehicle is sent to the vehicle. Based on the target pose of the vehicle-mounted drone, the current pose of the vehicle-mounted drone, and the driving-related data of the vehicle, a target control instruction is generated. After the vehicle-mounted drone executes the target control instruction, the vehicle-mounted drone is relatively stationary with the vehicle (moves following the vehicle). When the vehicle and the vehicle-mounted drone are relatively stationary and the vehicle-mounted drone is in the target pose, the vehicle-mounted drone can obtain a more comprehensive and accurate environmental image around the vehicle, thereby improving the comprehensiveness and accuracy of the obtained environmental image. The vehicle-mounted drone can realize real-time monitoring of the 360-degree full range around the vehicle body, effectively eliminating the viewing blind area of the vehicle-mounted camera, providing comprehensive and accurate environmental information around the vehicle body for the driver, and greatly improving the safety of vehicle driving and parking.

[0022] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0024] Figure 1 is a flowchart of a method for obtaining an environmental image in an embodiment of the present invention;

[0025] Figure 2 is a schematic structural diagram of an environmental image acquisition device in an embodiment of the present invention;

[0026] Figure 3 is a schematic structural diagram of a vehicle-mounted drone in an embodiment of the present invention;

[0027] Figure 4 is a schematic structural diagram of a monitoring system in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] To enable those skilled in the art to better understand the solution of the present invention, the following will clearly and completely describe the technical solution in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0030] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to users and the authorization of users should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0031] Embodiment 1

[0032] Figure 1 It is a flowchart of a method for acquiring an environmental image provided by an embodiment of the present invention. This embodiment is applicable to the situation of acquiring an environmental image. This method can be executed by the environmental image acquisition device in the embodiment of the present invention, and this device is configured in an on-vehicle drone, as Figure 1 shown. The method specifically includes the following steps:

[0033] S110, acquire the target pose and the current pose of the on-vehicle drone.

[0034] In this embodiment, the target pose is the desired pose of the on-vehicle drone relative to the vehicle, and the current pose is the current pose of the on-vehicle drone relative to the vehicle.

[0035] In this embodiment, the target pose includes: a target position and a target attitude. The target attitude includes: the attitude angles of the on-vehicle drone, and the attitude angles include: roll angle, pitch angle, and yaw angle. The target position includes: the specific positions of the drone in the horizontal and vertical directions.

[0036] In this embodiment, a drone landing gear and related connection devices are installed at a preset position on the vehicle roof to ensure the stable takeoff and landing and firm connection of the drone.

[0037] In this embodiment, the in-vehicle drone is equipped with a high-resolution camera, a stable gimbal, a wireless image transmission module, a positioning and navigation module, a vision recognition module, and a flight control module. The high-resolution camera is used to capture high-definition images around the vehicle body. The stable gimbal ensures the stability of the camera during flight. The stable gimbal mainly relies on an inertial measurement unit to sense the attitude changes of the in-vehicle drone. The gyroscope in the inertial measurement unit can measure the angular velocity of the gimbal, and the accelerometer can detect the linear acceleration of the gimbal. When the attitude of the drone changes, the inertial measurement unit quickly transmits this change information to the control system of the gimbal. The control system calculates the control commands required for the motor based on the received information and adjusts the attitude of the gimbal by driving the motor to keep the camera in a stable state. The wireless image transmission module is responsible for transmitting the captured images to the vehicle in real time (such as the in-vehicle display screen). The positioning and navigation module includes: a satellite navigation unit (such as GPS can be used), an inertial measurement unit, and a barometer. The inertial measurement unit is usually composed of sensors such as gyroscopes and accelerometers. The gyroscope is used to measure the angular velocity of the drone, and the attitude angle of the drone can be obtained through integral operation; the accelerometer is used to measure the acceleration of the drone in three axes, and the speed and displacement of the drone can be calculated by combining time information. The inertial measurement unit perceives the attitude changes and motion states of the drone in real time through fast sampling and calculation. The barometer indirectly measures the height of the drone by measuring the atmospheric pressure. Since the atmospheric pressure decreases regularly with the change of height, the barometer can calculate the height of the drone relative to the sea level or the takeoff point based on the measured atmospheric pressure value. The vision recognition module usually consists of a camera on the drone and related image processing algorithms. By analyzing the captured images, it identifies the feature points, markers, or texture information on the vehicle roof to determine the position and attitude of the drone relative to the vehicle. The flight control module controls the takeoff, landing, flight attitude, and speed of the drone.

[0038] Debug various parameters of the in-vehicle drone, including the resolution, frame rate, and shooting angle of the camera, the signal strength and stability of the wireless image transmission module, the accuracy and reliability of the positioning and navigation module, the flight parameters of the flight control module, the recognition accuracy of the vision recognition module, etc., to ensure the normal operation of the drone. For example, conduct a large number of image sample trainings on the vision recognition module so that it can accurately identify the stop position mark on the vehicle roof, and at the same time calibrate the positioning and navigation module to ensure the accuracy of height measurement.

[0039] It should be noted that after the vehicle starts, the driver starts the on-vehicle drone through the in-vehicle control button. The flight control module of the on-vehicle drone initializes the positioning and navigation module, the camera, the wireless image transmission module, etc., to ensure the normal operation of each module.

[0040] In this embodiment, the target pose may be a preset desired pose of the on-vehicle drone relative to the vehicle. The target pose may be pre-stored in the on-vehicle drone, or it may be that after the vehicle obtains the target pose, it sends the target pose to the on-vehicle drone. It should be noted that when the on-vehicle drone is in the desired pose above the vehicle, the on-vehicle drone can obtain a more comprehensive and accurate environmental image around the vehicle.

[0041] In this embodiment, the method for obtaining the current pose of the vehicle-mounted UAV can be as follows: When the vehicle is in a stationary state, send a flight instruction to the vehicle-mounted UAV (the flight instruction can be a vertical takeoff instruction), obtain the vehicle images collected during the execution of the flight instruction by the vehicle-mounted UAV, identify the vehicle images, obtain the target image coordinates of the ORB points of multiple angular contours of the vehicle, and determine the current pose of the vehicle-mounted UAV according to the target image coordinates of the ORB points of multiple angular contours of the vehicle and the actual coordinates of the ORB points of each angular contour of the vehicle. The method for obtaining the current pose of the vehicle-mounted UAV can also be as follows: When the vehicle is in a stationary state, send a flight instruction to the vehicle-mounted UAV (the flight instruction can be a vertical takeoff instruction), obtain the vehicle images collected during the execution of the flight instruction by the vehicle-mounted UAV, identify the vehicle images, obtain the target image coordinates of the centroids of multiple angular contours of the vehicle, and determine the current pose of the vehicle-mounted UAV according to the target image coordinates of the centroids of multiple angular contours of the vehicle and the actual coordinates of the centroids of multiple angular contours of the vehicle. The method for obtaining the current pose of the vehicle-mounted UAV can also be as follows: When the vehicle is in a stationary state, send a flight instruction to the vehicle-mounted UAV (the flight instruction can be a vertical takeoff instruction), obtain the vehicle images collected during the execution of the flight instruction by the vehicle-mounted UAV, identify the vehicle images, obtain the target image coordinates of the centroids of multiple angular contours of the vehicle, and determine the initial pose of the vehicle-mounted UAV according to the target image coordinates of the centroids of multiple angular contours of the vehicle and the actual coordinates of the centroids of multiple angular contours of the vehicle. If the pose deviation between the initial pose and the target pose of the vehicle-mounted UAV is less than the pose deviation threshold, then identify the vehicle images to obtain the target image coordinates of the ORB points of multiple angular contours of the vehicle; determine the current pose of the vehicle-mounted UAV according to the target image coordinates of the ORB points of multiple angular contours of the vehicle and the actual coordinates of the ORB points of each angular contour of the vehicle. The method for obtaining the current pose of the vehicle-mounted UAV can also be as follows: When the vehicle is in a driving state, generate a flight instruction according to the driving-related data of the vehicle and the GPS positioning information of the vehicle-mounted UAV, send the flight instruction to the vehicle-mounted UAV, obtain the vehicle images collected during the execution of the flight instruction by the vehicle-mounted UAV, identify the vehicle images, obtain the target image coordinates of the ORB points of multiple angular contours of the vehicle, and determine the current pose of the vehicle-mounted UAV according to the target image coordinates of the ORB points of multiple angular contours of the vehicle and the actual coordinates of the ORB points of each angular contour of the vehicle.The method for obtaining the current pose of the vehicle-mounted UAV can also be as follows: When the vehicle is in a driving state, a flight instruction is generated based on the driving-related data of the vehicle and the GPS positioning information of the vehicle-mounted UAV, the flight instruction is sent to the vehicle-mounted UAV, the vehicle images collected during the execution of the flight instruction by the vehicle-mounted UAV are obtained, the vehicle images are recognized to obtain the target image coordinates of the centroids of multiple angular contours of the vehicle, and the current pose of the vehicle-mounted UAV is determined according to the target image coordinates of the centroids of multiple angular contours of the vehicle and the actual coordinates of the centroids of multiple angular contours of the vehicle. The method for obtaining the current pose of the vehicle-mounted UAV can also be as follows: When the vehicle is in a driving state, a flight instruction is generated based on the driving-related data of the vehicle and the GPS positioning information of the vehicle-mounted UAV, the flight instruction is sent to the vehicle-mounted UAV, the vehicle images collected during the execution of the flight instruction by the vehicle-mounted UAV are obtained, the vehicle images are recognized to obtain the target image coordinates of the centroids of multiple angular contours of the vehicle, and the initial pose of the vehicle-mounted UAV is determined according to the target image coordinates of the centroids of multiple angular contours of the vehicle and the actual coordinates of the centroids of multiple angular contours of the vehicle. If the pose deviation between the initial pose and the target pose of the vehicle-mounted UAV is less than the pose deviation threshold, the vehicle images are recognized to obtain the target image coordinates of the ORB points of multiple angular contours of the vehicle; the current pose of the vehicle-mounted UAV is determined according to the target image coordinates of the ORB points of multiple angular contours of the vehicle and the actual coordinates of the ORB points of each angular contour of the vehicle.

[0042] It should be noted that the method for obtaining the current pose of the vehicle-mounted UAV can also be: receiving the current pose of the vehicle-mounted UAV sent by the vehicle. The method for the vehicle to determine the current pose of the vehicle-mounted UAV is the same as the method for the vehicle-mounted UAV to determine its own current pose, which will not be elaborated here.

[0043] Optionally, obtaining the current pose of the vehicle-mounted UAV includes:

[0044] Receiving the flight instruction sent by the vehicle and obtaining the target vehicle images collected during the execution of the flight instruction by the vehicle-mounted UAV.

[0045] In this embodiment, the flight instruction can be a vertical takeoff instruction or a flight instruction generated according to the GPS position information of the UAV and the driving-related data of the vehicle. The driving-related data of the vehicle includes data such as vehicle speed, acceleration, and GPS position information of the vehicle.

[0046] In this embodiment, if the vehicle is in a stationary state, the flight instruction can be a vertical takeoff instruction; if the vehicle is in a driving state (non-stationary state), the flight instruction can be generated according to the GPS position information of the UAV and the driving-related data of the vehicle.

[0047] In this embodiment, the method for obtaining the target vehicle image collected during the execution of the flight instruction by the in-vehicle drone may be: obtaining the altitude value during the execution of the flight instruction by the in-vehicle drone, and when the altitude data of the in-vehicle drone is greater than the preset altitude, obtaining the target vehicle image collected by the in-vehicle drone. In this embodiment, the method for obtaining the target vehicle image collected during the execution of the flight instruction by the in-vehicle drone may also be: obtaining the initial vehicle image collected during the execution of the flight instruction by the in-vehicle drone; identifying the initial vehicle image to obtain the position of the in-vehicle drone fixed bracket in the initial vehicle image; if the in-vehicle drone fixed bracket is within the central position area of the initial vehicle image, then using the initial vehicle image as the target vehicle image.

[0048] Identify the target vehicle image to obtain the target image coordinates of the centroids of multiple angular contours of the vehicle.

[0049] In this embodiment, the method for identifying the target vehicle image to obtain the target image coordinates of the centroids of multiple angular contours of the vehicle may be: performing edge detection on the target vehicle image to obtain the contour information of the vehicle, extracting the key feature points in the contour information as corner points, and calculating the image coordinates of the centroid based on the corner points and the contour area of the corner points.

[0050] In a specific example, to improve the subsequent recognition accuracy, it is necessary to preprocess the image first. Grayscale processing can be used to convert the color target vehicle image into a grayscale image to reduce the amount of data. Methods such as Gaussian filtering and median filtering can also be used to remove the noise in the image and avoid noise interference with subsequent contour and corner point detection. The contour of the vehicle is highlighted through the edge detection algorithm. Through steps such as Gaussian filtering to smooth the image, calculating the gradient magnitude and direction, non-maximum suppression, and double-threshold detection and connection, the edge of the vehicle is detected more accurately. Based on the edge detection result, the complete contour of the vehicle is obtained using the contour extraction algorithm. The corner point is a key feature point of the contour and can reflect the shape information of the vehicle. Based on the eigenvalue of the image grayscale matrix to screen the corner points, the corner points in the image are detected more effectively. For the detected corner points, the centroid can be calculated according to the contour area where they are located. After the above steps, the coordinates of the centroids of the angular contours have been calculated, and these coordinates can be output in a specific format according to the rules of the image coordinate system.

[0051] Determine the initial pose of the in-vehicle drone according to the target image coordinates of the centroids of multiple angular contours of the vehicle and the actual coordinates of the centroids of multiple angular contours of the vehicle.

[0052] In this embodiment, the way to obtain the actual coordinates of the centroids of multiple corner profiles of the vehicle can be: reading the actual coordinates of the centroids of multiple corner profiles of the vehicle from the design document of the vehicle. The way to obtain the actual coordinates of the centroids of multiple corner profiles of the vehicle can also be: calculating the actual coordinates of the centroids of multiple corner profiles of the vehicle according to the basic parameters in the design document of the vehicle.

[0053] In this embodiment, a camera model is established in advance: the camera model describes the mapping relationship between points in the three-dimensional world and points on the image plane. A common camera model is the pinhole camera model, which takes into account parameters such as the focal length and principal point of the camera. These parameters can be determined through camera calibration to obtain the internal parameter matrix of the camera and establish the projection relationship from actual coordinates to image coordinates. Then, solve the rotation and translation matrices: using at least three sets of corresponding points of image coordinates and actual coordinates, according to the camera model, through perspective transformation or more complex iterative algorithms (such as the least squares method), solve for the rotation matrix R and the translation vector t. The rotation matrix R describes the attitude of the on-vehicle drone, that is, the rotation angles around the three coordinate axes, and the translation vector t represents the position of the on-vehicle drone. Finally, determine the pose of the on-vehicle drone: the rotation matrix R and the translation vector t together constitute the pose of the on-vehicle drone. Thus, clarify the position and attitude of the on-vehicle drone relative to the vehicle. In practical applications, it may be affected by camera distortion, measurement errors, environmental factors, etc., and measures such as distortion correction, increasing the number of corresponding points, and filtering processing need to be taken to improve the accuracy and reliability of pose calculation.

[0054] If the pose deviation between the initial pose and the target pose of the on-vehicle drone is less than the pose deviation threshold, then identify the target vehicle image to obtain the target image coordinates of the ORB points of multiple corner profiles of the vehicle.

[0055] In this embodiment, the pose deviation is used to measure the degree of deviation between the initial pose and the target pose.

[0056] In this embodiment, if the pose deviation between the initial pose and the target pose of the on-vehicle drone is less than the pose deviation threshold, then the way to obtain the target image coordinates of the ORB points of multiple corner profiles of the vehicle by identifying the target vehicle image can be: if the pose deviation between the initial pose and the target pose of the on-vehicle drone is less than the pose deviation threshold, and the similarity between the target vehicle image and the reference vehicle image is greater than the similarity threshold, then identify the target vehicle image. It should be noted that the reference vehicle image is the vehicle image collected after pre-controlling the drone to be in the target pose.

[0057] In this embodiment, the method for identifying the target vehicle image to obtain the target image coordinates of the ORB points of multiple corner contours of the vehicle may be as follows: Use the Oriented FAST and Rotated BRIEF (ORB) algorithm to detect the ORB points in the target vehicle image, traverse all the ORB points, and determine whether they are located on the contour. If so, consider them as the ORB points on the corner contour. Read the target image coordinates of the ORB points on the corner contour.

[0058] Determine the current pose of the on-vehicle drone according to the target image coordinates of the ORB points of multiple corner contours of the vehicle and the actual coordinates of the ORB points of each corner contour of the vehicle.

[0059] In this embodiment, the method for determining the current pose of the on-vehicle drone according to the target image coordinates of the ORB points of multiple corner contours of the vehicle and the actual coordinates of the ORB points of each corner contour of the vehicle is similar to the method for determining the initial pose of the on-vehicle drone according to the target image coordinates of the centroids of multiple corner contours of the vehicle and the actual coordinates of the centroids of multiple corner contours of the vehicle, and will not be elaborated here.

[0060] Optionally, obtaining the target vehicle image collected during the execution of the flight instruction by the on-vehicle drone includes:

[0061] Obtain the initial vehicle image collected during the execution of the flight instruction by the on-vehicle drone.

[0062] In this embodiment, the initial vehicle image may be the vehicle image collected at a preset sampling period during the execution of the flight instruction by the on-vehicle drone. The initial vehicle image may also be the vehicle image collected during the execution of the flight instruction and after the on-vehicle drone ascends a preset height.

[0063] Identify the initial vehicle image to obtain the position of the on-vehicle drone fixed bracket in the initial vehicle image.

[0064] In this embodiment, the setting position of the on-vehicle drone fixed bracket needs to comprehensively consider factors such as vehicle structure, drone usage requirements, and safety. In the embodiment of the present invention, the on-vehicle drone fixed bracket is set on the roof of the vehicle. Its advantage is that the field of view is wide, and the drone is not easily blocked by the vehicle body during takeoff and landing, and a good shooting angle and flight space can be obtained. It is suitable for scenarios such as high-altitude shooting and road condition monitoring. It can be installed through a suction cup type or bolt fixed type bracket.

[0065] In this embodiment, the method for identifying the initial vehicle image to obtain the position of the in-vehicle drone fixing bracket in the initial vehicle image may be: inputting the initial vehicle image into a pre-trained target detection model to obtain the position of the in-vehicle drone fixing bracket in the initial vehicle image.

[0066] If the in-vehicle drone fixing bracket is within the central position area of the initial vehicle image, the initial vehicle image is used as the target vehicle image.

[0067] Optionally, it further includes:

[0068] If the in-vehicle drone fixing bracket is outside the central position area of the initial vehicle image and within the initial vehicle image, a first control instruction is generated according to the position of the in-vehicle drone fixing bracket in the initial vehicle image, the central position area of the initial vehicle image, and the driving-related data of the vehicle.

[0069] In this embodiment, the method for generating the first control instruction according to the position of the in-vehicle drone fixing bracket in the initial vehicle image, the central position area of the initial vehicle image, and the driving-related data of the vehicle may be: generating a flight path according to the position of the in-vehicle drone fixing bracket in the initial vehicle image, the central position area of the initial vehicle image, the current flight-related parameters of the in-vehicle drone, and the current driving-related parameters of the vehicle, and generating the first control instruction based on the flight path, the current flight-related parameters of the in-vehicle drone, and the current driving-related parameters of the vehicle.

[0070] In this embodiment, the first control instruction is used to control the in-vehicle drone to fly to the position of the in-vehicle drone fixing bracket in the central position area of the image in the captured vehicle image.

[0071] Obtain the image to be verified collected during the execution of the first control instruction by the in-vehicle drone.

[0072] In this embodiment, the vehicle image collected during the execution of the first control instruction by the in-vehicle drone is used as the image to be verified.

[0073] If the image to be verified passes the verification, the image to be verified is used as the target vehicle image.

[0074] In this embodiment, the method for verifying the image to be verified may be: if the in-vehicle drone fixing bracket is within the central position area of the image to be verified, the image to be verified passes the verification; if the in-vehicle drone fixing bracket is outside the central position area of the image to be verified, the image to be verified fails to pass the verification.

[0075] Optionally, it further includes:

[0076] Identify the initial vehicle image. If the on-vehicle drone fixing bracket does not exist in the initial vehicle image, obtain the flight-related data of the on-vehicle drone and the driving-related data of the vehicle.

[0077] In this embodiment, if the on-vehicle drone fixing bracket does not exist in the initial vehicle image, it indicates that the pose deviation between the on-vehicle drone and the vehicle is relatively large, and it is necessary to first control the on-vehicle drone to fly to a position area near the vehicle.

[0078] In this embodiment, the flight-related data of the on-vehicle drone includes: GPS positioning information, flight speed, flight direction and other information. The driving-related data of the vehicle includes: GPS positioning information of the vehicle, vehicle speed, acceleration and other information.

[0079] Generate a second control instruction according to the flight-related data of the on-vehicle drone and the driving-related data of the vehicle.

[0080] In this embodiment, the second control instruction is used to control the on-vehicle drone to fly to a position near the vehicle.

[0081] In this embodiment, the manner of generating the second control instruction according to the flight-related data of the on-vehicle drone and the driving-related data of the vehicle can be: generate a flight path according to the GPS positioning information of the on-vehicle drone, the GPS positioning information of the vehicle, the flight speed of the on-vehicle drone, the flight acceleration of the on-vehicle drone, the flight direction of the on-vehicle drone, the GPS positioning information of the vehicle, the vehicle speed of the vehicle, the acceleration of the vehicle and the driving direction of the vehicle, and generate a second control instruction according to the flight path of the on-vehicle drone, the flight-related data of the on-vehicle drone and the driving-related data of the vehicle.

[0082] Obtain the image to be verified collected during the on-vehicle drone flying and executing the second control instruction.

[0083] In this embodiment, use the vehicle image collected during the on-vehicle drone flying and executing the second control instruction as the image to be verified.

[0084] If the image to be verified passes the verification, use the image to be verified as the target vehicle image.

[0085] In this embodiment, the manner of verifying the image to be verified can be: if the on-vehicle drone fixing bracket is within the central position area of the image to be verified, the image to be verified passes the verification; if the on-vehicle drone fixing bracket is outside the central position area of the image to be verified, the image to be verified does not pass the verification.

[0086] Optionally, if the image to be verified passes the verification, using the image to be verified as the target vehicle image includes:

[0087] Identifying the image to be verified to obtain the position of the on-vehicle UAV fixing bracket in the image to be verified.

[0088] If the on-vehicle UAV fixing bracket is within the central position area of the image to be verified, the image to be verified passes the verification, and the image to be verified is used as the target vehicle image.

[0089] In this embodiment, if the on-vehicle UAV fixing bracket is within the central position area of the image to be verified, the image to be verified passes the verification, and the image to be verified is used as the target vehicle image; if the on-vehicle UAV fixing bracket is outside the central position area of the image to be verified, the image to be verified fails to pass the verification.

[0090] Optionally, it further includes:

[0091] If the pose deviation between the initial pose and the target pose of the on-vehicle UAV is greater than or equal to the pose deviation threshold, a third control instruction is generated according to the initial pose of the on-vehicle UAV, the target pose, and the driving-related data of the vehicle.

[0092] In this embodiment, the third control instruction is used to control the on-vehicle UAV to be in the target pose.

[0093] In this embodiment, the method of generating the third control instruction according to the initial pose of the on-vehicle UAV, the target pose, and the driving-related data of the vehicle may be: generating the third control instruction according to the initial pose of the on-vehicle UAV, the target pose, the flight-related parameters of the on-vehicle UAV, and the driving-related data of the vehicle. After the on-vehicle UAV executes the third control instruction, the current pose of the on-vehicle UAV is determined according to the first vehicle image collected during the flight of the on-vehicle UAV.

[0094] In this embodiment, the method for determining the current pose of the vehicle-mounted UAV according to the first vehicle image collected during the flight of the vehicle-mounted UAV may be as follows: After the vehicle-mounted UAV executes the third control instruction, it is assumed that the pose deviation between the vehicle-mounted UAV and the target pose is small enough, and then the current pose of the vehicle-mounted UAV is determined according to the first vehicle image collected by the vehicle-mounted UAV. The method for determining the current pose of the vehicle-mounted UAV according to the first vehicle image collected during the flight of the vehicle-mounted UAV may also be as follows: After the vehicle-mounted UAV executes the third control instruction, it is necessary to verify whether the pose deviation between the vehicle-mounted UAV and the target pose is small enough. If the verification is passed, the current pose of the vehicle-mounted UAV is determined according to the first vehicle image collected by the vehicle-mounted UAV.

[0095] Optionally, determining the current pose of the vehicle-mounted UAV according to the first vehicle image collected during the flight of the vehicle-mounted UAV includes:

[0096] Identifying the first image coordinates of the ORB points of multiple angular contours of the vehicle from the first vehicle image collected during the flight of the vehicle-mounted UAV.

[0097] Determining the current pose of the vehicle-mounted UAV according to the first image coordinates of the ORB points of multiple angular contours of the vehicle and the actual coordinates of the ORB points of each angular contour of the vehicle.

[0098] In this embodiment, it is assumed that the pose deviation between the vehicle-mounted UAV and the target pose is small enough, and the current pose of the vehicle-mounted UAV is directly determined according to the first vehicle image collected by the vehicle-mounted UAV.

[0099] In this embodiment, determining the current pose of the vehicle-mounted UAV according to the first image coordinates of the ORB points of multiple angular contours of the vehicle and the actual coordinates of the ORB points of each angular contour of the vehicle is similar to the method for determining the current pose of the vehicle-mounted UAV described above, and will not be elaborated here.

[0100] Optionally, determining the current pose of the vehicle-mounted UAV according to the first vehicle image collected during the flight of the vehicle-mounted UAV includes:

[0101] Identifying the first image coordinates of the centroids of multiple angular contours of the vehicle from the first vehicle image collected during the flight of the vehicle-mounted UAV.

[0102] Updating the initial pose of the vehicle-mounted UAV according to the first image coordinates of the centroids of multiple angular contours of the vehicle and the actual coordinates of the centroids of multiple angular contours of the vehicle.

[0103] If the pose deviation between the initial pose of the updated in-vehicle UAV and the target pose is less than the pose deviation threshold, then identify the first vehicle image to obtain the first image coordinates of the ORB points of multiple angular contours of the vehicle.

[0104] Determine the current pose of the in-vehicle UAV according to the first image coordinates of the ORB points of multiple angular contours of the vehicle and the actual coordinates of the ORB points of each angular contour of the vehicle.

[0105] In this embodiment, after the in-vehicle UAV executes the third control instruction, it is necessary to verify whether the pose deviation between the in-vehicle UAV and the target pose is small enough. The following method is used to verify whether the pose deviation between the in-vehicle UAV and the target pose is small enough: identify the first vehicle image collected during the flight of the in-vehicle UAV to obtain the first image coordinates of the centroids of multiple angular contours of the vehicle; update the initial pose of the in-vehicle UAV according to the first image coordinates of the centroids of multiple angular contours of the vehicle and the actual coordinates of the centroids of multiple angular contours of the vehicle; if the pose deviation between the updated initial pose of the in-vehicle UAV and the target pose is less than the pose deviation threshold, then determine that the pose deviation between the in-vehicle UAV and the target pose is small enough.

[0106] S120, generate a target control instruction according to the target pose of the in-vehicle UAV, the current pose of the in-vehicle UAV, and the driving-related data of the vehicle.

[0107] In this embodiment, the target control instruction includes at least one of: a flight attitude-related control instruction, a flight speed-related control instruction, and a flight acceleration-related control instruction. After the in-vehicle UAV executes the target control instruction, the in-vehicle UAV can accurately hover above the roof of the vehicle, be in the target pose, and move with the vehicle.

[0108] In this embodiment, the driving-related data of the vehicle includes: vehicle speed and acceleration. It should be noted that both the vehicle speed and acceleration are vectors and carry direction information.

[0109] In this embodiment, the method for generating a target control instruction based on the target pose of the vehicle-mounted UAV, the current pose of the vehicle-mounted UAV, and the driving-related data of the vehicle may be as follows: determining the target flight path of the vehicle-mounted UAV according to the target pose of the vehicle-mounted UAV, the current pose of the vehicle-mounted UAV, the vehicle speed, and the vehicle acceleration; and generating a target control instruction according to the target flight path of the vehicle-mounted UAV. The method for generating a target control instruction based on the target pose of the vehicle-mounted UAV, the current pose of the vehicle-mounted UAV, and the driving-related data of the vehicle may also be as follows: determining the target flight path of the vehicle-mounted UAV according to the target pose and the current pose of the vehicle-mounted UAV; and generating a target control instruction according to the target flight path of the vehicle-mounted UAV, the vehicle speed, and the vehicle acceleration.

[0110] Optionally, the driving-related data of the vehicle includes vehicle speed and acceleration.

[0111] Generating a target control instruction according to the target pose of the vehicle-mounted UAV, the current pose of the vehicle-mounted UAV, and the driving-related data of the vehicle includes:

[0112] Determining the target flight path of the vehicle-mounted UAV according to the target pose of the vehicle-mounted UAV, the current pose of the vehicle-mounted UAV, the vehicle speed, and the vehicle acceleration.

[0113] In this embodiment, both the vehicle speed and the vehicle acceleration are real-time acquired data.

[0114] Generating a target control instruction according to the target flight path of the vehicle-mounted UAV, the vehicle speed, and the vehicle acceleration.

[0115] In this embodiment, the method for generating a target control instruction according to the target flight path of the vehicle-mounted UAV, the vehicle speed, and the vehicle acceleration may be as follows: determining the speed and acceleration of the vehicle-mounted UAV according to the target flight path of the vehicle-mounted UAV, the vehicle speed, and the vehicle acceleration, and generating a target control instruction according to the speed and acceleration of the vehicle-mounted UAV.

[0116] The embodiment of the present invention improves the accuracy and stability of the positioning of the vehicle-mounted UAV, enabling it to accurately maintain its position and attitude even in complex environments, providing a more reliable guarantee for real-time monitoring around the vehicle body.

[0117] S130, after the vehicle-mounted UAV executes the target control instruction, sending an environmental image around the vehicle to the vehicle.

[0118] In this embodiment, after the in-vehicle drone executes the target control instruction, the way to send the environmental image around the vehicle to the vehicle may be: after the in-vehicle drone executes the target control instruction, obtain the environmental image around the vehicle collected during the flight of the in-vehicle drone, and send the collected environmental image around the vehicle to the vehicle.

[0119] In this embodiment, after the vehicle receives the environmental image around the vehicle, it can display the environmental image around the vehicle through a display screen, so that the driver can understand the situation around the vehicle body in real time, discover potential safety risks in advance, take corresponding driving measures in time, avoid the occurrence of collision accidents, and ensure the life and property safety of the passengers and drivers. It can also analyze the environmental image around the vehicle to obtain an assisted driving decision and execute the assisted driving decision.

[0120] It should be noted that in a narrow parking lot or a complex parking environment, the driver can rely on the environmental image around the vehicle sent by the in-vehicle drone to more accurately judge the distance and position relationship between the vehicle and the surrounding obstacles, easily complete the parking operation, and improve the convenience and safety of parking.

[0121] Optionally, after the in-vehicle drone executes the target control instruction, sending the environmental image around the vehicle to the vehicle includes:

[0122] After the in-vehicle drone executes the target control instruction, obtain the image to be verified collected during the flight of the in-vehicle drone.

[0123] In this embodiment, the image to be verified is the vehicle image collected during the flight state of the in-vehicle drone after the in-vehicle drone executes the target control instruction.

[0124] If the image to be verified passes the verification, obtain the environmental image around the vehicle and send the environmental image around the vehicle to the vehicle.

[0125] In this embodiment, the method for verifying the to-be-verified image may be as follows: Identify the to-be-verified image to obtain the position of the vehicle-mounted drone fixing bracket in the initial vehicle image; if the vehicle-mounted drone fixing bracket is within the central position area of the initial vehicle image, the to-be-verified image passes the verification; if the vehicle-mounted drone fixing bracket is outside the central position area of the initial vehicle image, the to-be-verified image fails the verification. The method for verifying the to-be-verified image may also be as follows: Identify the to-be-verified image to obtain the target image coordinates of the centroid of multiple angular contours of the vehicle; determine the initial pose of the vehicle-mounted drone according to the target image coordinates of the centroid of multiple angular contours of the vehicle and the actual coordinates of the centroid of multiple angular contours of the vehicle; if the pose deviation between the initial pose of the vehicle-mounted drone and the target pose is less than a first value, the to-be-verified image passes the verification. The first value is less than the pose deviation threshold. If the pose deviation between the initial pose of the vehicle-mounted drone and the target pose is greater than or equal to the first value, the to-be-verified image fails the verification. The method for verifying the to-be-verified image may also be as follows: Identify the to-be-verified image to obtain the target image coordinates of the ORB points of multiple angular contours of the vehicle; determine the current pose of the vehicle-mounted drone according to the target image coordinates of the ORB points of multiple angular contours of the vehicle and the actual coordinates of the ORB points of each angular contour of the vehicle; if the pose deviation between the current pose of the vehicle-mounted drone and the target pose is less than the first value, the to-be-verified image passes the verification; if it is greater than or equal to the first value, the to-be-verified image fails the verification.

[0126] In this embodiment, after the vehicle-mounted drone executes the target control instruction, the collected image is verified. In the case of passing the verification, the environmental image around the vehicle collected is sent to the vehicle so that the vehicle can receive a more comprehensive and accurate environmental image. [[ID=][4]

[0127] Optionally, the vehicle includes: a display screen;

[0128] [[ID=][9]The environmental image acquisition method further includes:

[0129] Send the environmental image around the vehicle to the display screen of the vehicle so that the display screen displays the environmental image around the vehicle.

[0130] In this embodiment, the in-vehicle display screen is installed at a preset position in front of the driver's seat and connected and set with the wireless image transmission module of the in-vehicle drone to ensure that the display screen can accurately receive and display the images transmitted by the in-vehicle drone. The vehicle control unit is programmed and configured to communicate with the drone and implement various control operations on the drone. At the same time, the monitoring strategy of the in-vehicle drone is automatically adjusted according to the driving-related data of the vehicle. For example, the communication protocol between the vehicle control unit and the in-vehicle drone is set to ensure the accuracy and timeliness of data transmission. At the same time, an algorithm is written to enable the vehicle control unit to automatically adjust the flight altitude and shooting angle of the drone according to information such as the speed and steering of the vehicle.

[0131] In this embodiment, the in-vehicle display screen: is installed in front of the driver's seat in the vehicle for the driver to view conveniently. The display screen is connected to the wireless image transmission module of the in-vehicle drone to receive and display the real-time images around the vehicle body captured by the drone. The vehicle control unit: is communicatively connected to the in-vehicle drone and can control various operations of the drone, such as starting, stopping, adjusting the flight altitude, and switching the shooting angle. At the same time, the vehicle control unit can also automatically adjust the monitoring strategy of the drone according to the driving state of the vehicle (such as speed, steering, etc.) to ensure the effectiveness and pertinence of the monitoring.

[0132] In a specific example, the in-vehicle drone takes off vertically from the roof of the vehicle under the guidance of the positioning and navigation module. During the takeoff process, the positioning and navigation module preliminarily determines the position and altitude of the in-vehicle drone using satellite signals, causing the in-vehicle drone to rise to a certain altitude (such as 5 - 8 meters above the roof). At this time, the visual recognition module takes pictures of the surrounding environment through the high-resolution camera on the in-vehicle drone, and uses the image feature extraction algorithm to identify the feature points of the vehicle (such as the car contour, the position where the in-vehicle drone is parked on the roof, etc.). Based on these feature points, the visual positioning algorithm is used to calculate the current pose of the drone relative to the vehicle. The calculated current pose is fused with the preliminary positioning result of the positioning and navigation module to calibrate the position and attitude of the drone, ensuring that the drone can accurately hover at a preset position above the roof of the vehicle. During the hovering process, the visual recognition module continues to work, continuously monitoring changes in the surrounding environment. When it detects that the pose of the in-vehicle drone deviates due to environmental changes, it timely sends an adjustment instruction to the flight control module. The flight control module adjusts the motor speed and blade angle of the drone to maintain the stable hovering of the in-vehicle drone. Image acquisition and transmission: The high-resolution camera of the in-vehicle drone takes real-time pictures of the environment within a 360-degree range around the vehicle body at a certain frame rate (such as 30 frames per second). After the taken pictures are stabilized by the stable gimbal, they are transmitted in real-time in the form of wireless signals to the receiving module of the in-vehicle display screen by the wireless image transmission module. Image display and monitoring: After receiving the image signal transmitted by the drone, the in-vehicle display screen analyzes and processes it, and then displays the clear real-time image around the vehicle body. The driver can observe the display screen to understand the situations of pedestrians, vehicles, obstacles, etc. around the vehicle body in real-time, make driving decisions in advance, and avoid the occurrence of collision accidents.

[0133] During the vehicle's driving process, the vehicle control unit continuously obtains information such as the driving speed and direction of the vehicle, and transmits this information to the flight control module of the drone. The visual recognition module continuously conducts visual perception of the vehicle's surrounding environment. By identifying the fixed feature points (such as road signs, street lights, etc.) in the vehicle's surrounding environment and the features of the vehicle itself (such as the body contour, headlight position, etc.), it calculates the pose change of the drone relative to the vehicle in real-time. The flight control module automatically adjusts the flight speed, direction, and attitude of the in-vehicle drone according to the vehicle driving information transmitted by the vehicle control unit and the pose change information calculated by the visual perception pose positioning module, so that the in-vehicle drone always follows the vehicle's movement and maintains continuous monitoring of the environment around the vehicle body. For example, when the vehicle turns, the visual recognition module detects the change in the steering angle and rate of the vehicle, combines with the surrounding environmental features, calculates that the in-vehicle drone needs to adjust its flight direction and attitude, and the flight control module controls the in-vehicle drone to turn synchronously according to this information, ensuring that the relative position between the in-vehicle drone and the vehicle remains stable and continuously providing accurate monitoring images around the vehicle body.

[0134] Optionally, after sending the environmental image around the vehicle to the vehicle, it further includes:

[0135] Receiving a landing instruction sent by the vehicle;

[0136] Obtaining a second vehicle image and altitude data collected during the process of the in-vehicle drone executing the landing instruction.

[0137] In this embodiment, the altitude data is the altitude data collected by a barometer.

[0138] Identifying the second vehicle image to obtain the position information of the in-vehicle drone fixing bracket.

[0139] Updating the landing instruction according to the position information of the in-vehicle drone fixing bracket and the altitude data;

[0140] After executing the updated landing instruction, the in-vehicle drone lands on the in-vehicle drone fixing bracket.

[0141] When the vehicle arrives at the destination and stops, the driver controls the in-vehicle drone to land through the in-vehicle control button. The in-vehicle drone starts to descend under the control of the flight control module. During the descent, the vision recognition module continuously collects and analyzes the image below to identify the position information of the in-vehicle drone fixing bracket. At the same time, the barometer measures the altitude of the drone in real time and transmits the altitude data to the flight control module. The flight control module accurately adjusts the flight attitude and descent speed of the drone according to the position information of the in-vehicle drone fixing bracket feedback by the vision recognition module and the altitude data of the barometer, so that the drone slowly and smoothly lands on the in-vehicle drone fixing bracket and then stops working. The drone can accurately identify the in-vehicle drone fixing bracket according to the image, realizing precise landing, reducing the risk of drone damage caused by inaccurate landing, and at the same time improving the usability and stability.

[0142] In this embodiment, by controlling the flight attitude and shooting angle of the in-vehicle drone, the driver can flexibly adjust the monitoring perspective according to actual needs, focus on key areas around the vehicle, such as the turning direction, the rear of reversing, etc., and improve the pertinence and effectiveness of monitoring.

[0143] During actual use, the driver can flexibly operate the on-vehicle drone for monitoring around the vehicle body according to different driving scenarios and requirements. At the same time, based on actual use feedback and test results, the system is optimized and improved, such as further improving the flight stability and image transmission quality of the drone, optimizing the visual recognition algorithm to improve the accuracy and speed of stop position recognition, optimizing the control algorithm of the vehicle control unit, etc., continuously enhancing the performance and reliability of the system. For example, tests are carried out under different lighting conditions and weather conditions, the recognition data of the visual recognition module is collected, and the recognition algorithm is optimized to enable it to adapt to various complex environments.

[0144] The technical solution of this embodiment is to obtain the target pose and the current pose of the on-vehicle drone, where the target pose is the expected pose of the on-vehicle drone relative to the vehicle, and the current pose is the current pose of the on-vehicle drone relative to the vehicle; generate a target control instruction according to the target pose of the on-vehicle drone, the current pose of the on-vehicle drone, and the driving-related data of the vehicle; after the on-vehicle drone executes the target control instruction, obtain the environmental image around the vehicle and send the environmental image around the vehicle to the vehicle. Based on the target pose of the on-vehicle drone, the current pose of the on-vehicle drone, and the driving-related data of the vehicle, a target control instruction is generated. After the on-vehicle drone executes the target control instruction, the on-vehicle drone is relatively stationary with the vehicle (follows the vehicle to move). When the vehicle and the on-vehicle drone are relatively stationary and the on-vehicle drone is in the target pose, the on-vehicle drone can obtain a more comprehensive and accurate environmental image around the vehicle, thereby improving the comprehensiveness and accuracy of the obtained environmental image.

[0145] Embodiment 2

[0146] Figure 2 It is a schematic structural diagram of an environmental image acquisition device provided by an embodiment of the present invention. This embodiment is applicable to the situation of environmental image acquisition. The device can be implemented in a software and / or hardware manner, and the device can be integrated in any on-vehicle drone that provides the function of environmental image acquisition, such as Figure 2 As shown, the environmental image acquisition device specifically includes: an acquisition module 210, a target control instruction generation module 220, and an environmental image sending module 230.

[0147] Among them, the acquisition module is used to obtain the target pose and the current pose of the on-vehicle drone, where the target pose is the expected pose of the on-vehicle drone relative to the vehicle, and the current pose is the current pose of the on-vehicle drone relative to the vehicle;

[0148] A target control instruction generation module, configured to generate a target control instruction according to the target pose of the vehicle-mounted unmanned aerial vehicle, the current pose of the vehicle-mounted unmanned aerial vehicle, and the driving-related data of the vehicle;

[0149] An environmental image sending module, configured to obtain an environmental image around the vehicle after the vehicle-mounted unmanned aerial vehicle executes the target control instruction, and send the environmental image around the vehicle to the vehicle.

[0150] The above product can execute the method provided in any embodiment of the present invention, and has a functional module and beneficial effects corresponding to the execution of the method.

[0151] Embodiment III

[0152] Figure 3 FIG. shows a schematic structural diagram of a vehicle-mounted unmanned aerial vehicle 10 that can be used to implement an embodiment of the present invention. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0153] As Figure 3 shown, the vehicle-mounted unmanned aerial vehicle 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. Among them, the memory stores a computer program executable by the at least one processor, and the processor 11 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the vehicle-mounted unmanned aerial vehicle 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0154] Multiple components in the vehicle-mounted unmanned aerial vehicle 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the vehicle-mounted unmanned aerial vehicle 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0155] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the environmental image acquisition method.

[0156] In some embodiments, the environmental image acquisition method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the vehicle-mounted drone 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the environmental image acquisition method described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the environmental image acquisition method by any other suitable means (e.g., by means of firmware).

[0157] The various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor, and can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0158] The computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowchart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, as an independent software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0159] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0160] To provide for interaction with a user, the systems and techniques described herein can be implemented on an on-vehicle drone that has: a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the on-vehicle drone. Other kinds of devices can also be used to provide for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).

[0161] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of the communication network include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0162] A computing system may include a client and a server. The client and the server are generally far from each other and usually interact via a communication network. The relationship between the client and the server is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, solving the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0163] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is made herein.

[0164] An embodiment of the present invention also provides a computer program product, including a computer program which, when executed by a processor, implements the environmental image acquisition method according to any embodiment of the present invention.

[0165] In the process of implementing the computer program product, the computer program code for performing the operations of the present invention can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN) - or can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet). [[ID=eleven]]

[0166] Embodiment Four

[0167] Figure 4 A schematic structural diagram of a monitoring system 40 that can be used to implement the embodiments of the present invention is shown. The monitoring system 40 includes: an in-vehicle drone 410 and a vehicle 420. The in-vehicle drone is used to execute the various methods and processes described above, such as the environmental image acquisition method.

[0168] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub - combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An environmental image acquisition method, characterized in that, Applied to an in-vehicle drone, the environmental image acquisition method includes: Obtaining the target pose and the current pose of the in-vehicle drone, where the target pose is the desired pose of the in-vehicle drone relative to the vehicle, and the current pose is the current pose of the in-vehicle drone relative to the vehicle; Generating a target control instruction according to the target pose of the in-vehicle drone, the current pose of the in-vehicle drone, and the driving-related data of the vehicle; After the in-vehicle drone executes the target control instruction, obtaining the environmental image around the vehicle and sending the environmental image around the vehicle to the vehicle.

2. The method according to claim 1, wherein Obtaining the current pose of the in-vehicle drone includes: Receiving the flight instruction sent by the vehicle and obtaining the target vehicle image collected during the in-vehicle drone's execution of the flight instruction; Identifying the target vehicle image to obtain the target image coordinates of the centroids of multiple angular contours of the vehicle; Determining the initial pose of the in-vehicle drone according to the target image coordinates of the centroids of multiple angular contours of the vehicle and the actual coordinates of the centroids of multiple angular contours of the vehicle; If the pose deviation between the initial pose and the target pose of the in-vehicle drone is less than the pose deviation threshold, identifying the target vehicle image to obtain the target image coordinates of the ORB points of multiple angular contours of the vehicle; Determining the current pose of the in-vehicle drone according to the target image coordinates of the ORB points of multiple angular contours of the vehicle and the actual coordinates of the ORB points of each angular contour of the vehicle.

3. The method according to claim 2, wherein Obtaining the target vehicle image collected during the in-vehicle drone's execution of the flight instruction includes: Obtaining the initial vehicle image collected during the in-vehicle drone's execution of the flight instruction; Identifying the initial vehicle image to obtain the position of the in-vehicle drone's fixed bracket in the initial vehicle image; If the in-vehicle drone's fixed bracket is within the central position area of the initial vehicle image, taking the initial vehicle image as the target vehicle image.

4. The method according to claim 3, characterized in that It further includes: If the in-vehicle drone's fixed bracket is outside the central position area of the initial vehicle image and within the initial vehicle image, generating a first control instruction according to the position of the in-vehicle drone's fixed bracket in the initial vehicle image, the central position area of the initial vehicle image, and the driving-related data of the vehicle; Obtaining the image to be verified collected during the in-vehicle drone's execution of the first control instruction; If the image to be verified passes the verification, taking the image to be verified as the target vehicle image.

5. The method according to claim 3, wherein It further includes: Identifying the initial vehicle image. If the in-vehicle drone's fixed bracket does not exist in the initial vehicle image, obtaining the flight-related data of the in-vehicle drone and the driving-related data of the vehicle; Generating a second control instruction according to the flight-related data of the in-vehicle drone and the driving-related data of the vehicle; Obtaining the image to be verified collected during the in-vehicle drone's flight execution of the second control instruction; If the image to be verified passes the verification, taking the image to be verified as the target vehicle image.

6. The method according to claim 4 or 5, characterized in that If the image to be verified passes the verification, use the image to be verified as the target vehicle image, including: Identify the image to be verified to obtain the position of the on-vehicle UAV fixing bracket in the image to be verified; If the on-vehicle UAV fixing bracket is within the central position area of the image to be verified, the image to be verified passes the verification, and the image to be verified is used as the target vehicle image.

7. The method according to claim 2, wherein It also includes: If the pose deviation between the initial pose and the target pose of the on-vehicle UAV is greater than or equal to the pose deviation threshold, generate a third control instruction according to the initial pose of the on-vehicle UAV, the target pose, and the driving-related data of the vehicle; After the on-vehicle UAV executes the third control instruction, determine the current pose of the on-vehicle UAV according to the first vehicle image collected during the flight of the on-vehicle UAV.

8. The method according to claim 7, characterized in that Determine the current pose of the on-vehicle UAV according to the first vehicle image collected during the flight of the on-vehicle UAV, including: Identify the first vehicle image collected during the flight of the on-vehicle UAV to obtain the first image coordinates of the ORB points of multiple angular contours of the vehicle; Determine the current pose of the on-vehicle UAV according to the first image coordinates of the ORB points of multiple angular contours of the vehicle and the actual coordinates of the ORB points of each angular contour of the vehicle.

9. The method according to claim 7, characterized in that, Determine the current pose of the on-vehicle UAV according to the first vehicle image collected during the flight of the on-vehicle UAV, including: Identify the first vehicle image collected during the flight of the on-vehicle UAV to obtain the first image coordinates of the centroids of multiple angular contours of the vehicle; Update the initial pose of the on-vehicle UAV according to the first image coordinates of the centroids of multiple angular contours of the vehicle and the actual coordinates of the centroids of multiple angular contours of the vehicle; If the pose deviation between the updated initial pose of the on-vehicle UAV and the target pose is less than the pose deviation threshold, identify the first vehicle image to obtain the first image coordinates of the ORB points of multiple angular contours of the vehicle; Determine the current pose of the on-vehicle UAV according to the first image coordinates of the ORB points of multiple angular contours of the vehicle and the actual coordinates of the ORB points of each angular contour of the vehicle.

10. The method according to claim 1, characterized in that The driving-related data of the vehicle includes: vehicle speed and acceleration; Generate a target control instruction according to the target pose of the on-vehicle UAV, the current pose of the on-vehicle UAV, and the driving-related data of the vehicle, including: Determine the target flight path of the on-vehicle UAV according to the target pose of the on-vehicle UAV, the current pose of the on-vehicle UAV, the vehicle speed of the vehicle, and the acceleration of the vehicle; Generate a target control instruction according to the target flight path of the on-vehicle UAV, the vehicle speed of the vehicle, and the acceleration of the vehicle.

11. The method according to claim 1, characterized in that, After the on-vehicle UAV executes the target control instruction, obtain the environmental image around the vehicle and send the environmental image around the vehicle to the vehicle, including: After the on-vehicle UAV executes the target control instruction, obtain the image to be verified collected during the flight of the on-vehicle UAV; If the image to be verified passes the verification, obtain the environmental image around the vehicle and send the environmental image around the vehicle to the vehicle.

12. The method according to claim 1, wherein The vehicle includes: a display screen; The environmental image acquisition method further includes: Send the environmental image around the vehicle to the display screen of the vehicle so that the display screen displays the environmental image around the vehicle.

13. The method according to claim 1, characterized in that, After sending the environmental image around the vehicle to the vehicle, it further includes: Receive the landing instruction sent by the vehicle; Obtain the second vehicle image and altitude data collected during the vehicle-mounted drone's execution of the landing instruction; Identify the second vehicle image to obtain the position information of the vehicle-mounted drone fixing bracket; Update the landing instruction according to the position information of the vehicle-mounted drone fixing bracket and the altitude data; After executing the updated landing instruction, the vehicle-mounted drone lands on the vehicle-mounted drone fixing bracket.

14. An environmental image acquisition device, characterized in that, Configured in the vehicle-mounted drone, the environmental image acquisition device includes: An acquisition module for acquiring the target pose of the vehicle-mounted drone and the current pose of the vehicle-mounted drone, where the target pose is the desired pose of the vehicle-mounted drone relative to the vehicle, and the current pose is the current pose of the vehicle-mounted drone relative to the vehicle; A target control instruction generation module for generating a target control instruction according to the target pose of the vehicle-mounted drone, the current pose of the vehicle-mounted drone, and the driving-related data of the vehicle; An environmental image sending module for obtaining the environmental image around the vehicle and sending the environmental image around the vehicle to the vehicle after the vehicle-mounted drone executes the target control instruction.

15. An in-vehicle drone, characterized in that, The vehicle-mounted drone includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the environmental image acquisition method according to any one of claims 1-13.

16. A monitoring system, characterized in that, The monitoring system includes: a vehicle-mounted drone and a vehicle, and the vehicle-mounted drone is used to execute the environmental image acquisition method according to any one of claims 1-13.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a processor to implement the environmental image acquisition method according to any one of claims 1-13 when executed.

18. A computer program product, characterized in that, The computer program product includes a computer program that implements the environmental image acquisition method according to any one of claims 1-13 when executed by a processor.

Images