A method for processing unmanned aerial vehicle and wearable device perception data

Abstract

Embodiments of the present application relate to a method for processing unmanned aerial vehicle and wearable device sensing data, the method comprising: a first server receiving first sensing data sent by a first unmanned aerial vehicle; receiving second sensing data sent by a first wearable device; generating a corresponding first real scene map according to a first body coordinate, a first body height, a first bird's eye view and a preset high-precision map for bird's eye view real scene map construction; generating a corresponding overall AR map and a personal AR map according to a first device coordinate, a first flat view and the first real scene map for AR map construction; displaying the overall AR map; and sending the personal AR map to the first wearable device. The present application can improve the timeliness of background scheduling, and can improve the action efficiency and cooperation efficiency of the front end.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a method for processing sensing data of a UAV and a wearable device. BACKGROUND

[0002] Conventional command and dispatch systems mostly dispatch personnel based on walkie-talkies with positioning devices. This dispatching method based on voice communication is often insufficient in more complex multi-party joint dispatch scenarios. The main reasons are as follows: 1) the background dispatch personnel cannot directly see the real scene state at the front end, which may lead to problems such as untimely dispatching; 2) the front action personnel cannot receive intuitive map push at the front end, and cannot accurately identify the identities of the surrounding personnel in time, which may also lead to problems such as slow action at the front and poor multi-party cooperation. SUMMARY

[0003] The present application aims to solve the defects of the prior art, and provides a method for processing sensing data of a UAV and a wearable device, an electronic device and a computer readable storage medium. The method comprises the following steps: configuring a UAV for the dispatch scene to take bird's-eye view images to generate corresponding UAV sensing data, and transmitting the UAV sensing data to a rear server; configuring a wearable device for a front action personnel to take close-range environment images to generate corresponding device sensing data, and transmitting the device sensing data to the rear server; creating a real scene map based on the UAV sensing data and a high-precision map by the rear server, and creating an Augmented Reality (AR) object on the real scene map based on the target recognition result of the device sensing data and the building and road semantic information of the high-precision map to obtain a corresponding overall AR map, and cutting the AR map based on the position of each wearable device to obtain a corresponding individual AR map; displaying the overall AR map by the rear server to help the background dispatch personnel know the real scene state at the front end in the first time, and pushing the individual AR map to each wearable device by the rear server to help the front action personnel understand the environment state around them and identify the identities of the surrounding personnel in the first time. Through the present application, the timeliness of the background dispatch can be improved, and the action efficiency and cooperation efficiency at the front end can be improved.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application provides a method for processing sensing data of a UAV and a wearable device, which comprises the following steps:

[0005] A first server receives first sensing data sent by a first UAV; the first sensing data comprises a first body coordinate, a first body height and a first bird's-eye view image;

[0006] Receiving second sensing data sent by a first wearable device: the second sensing data comprises a first device coordinate and a first heads-up view image;

[0007] According to the first body coordinates, the first body height, the first bird's eye view and a preset high-precision map, a corresponding first real-scene map is constructed by bird's eye real-scene map construction;

[0008] According to the first device coordinates, the first overhead view and the first real-scene map, a corresponding overall AR map and a personal AR map are constructed by AR map construction;

[0009] The overall AR map is displayed, and the personal AR map is sent to the first wearable device.

[0010] Preferably, the first server is connected with the first unmanned aerial vehicle and the first wearable device through a wireless communication mode; the wireless communication mode includes 4G, 5G and Wi Fi communication modes;

[0011] The first unmanned aerial vehicle includes a first positioning module and a first bird's eye camera; the first positioning module includes a first GPS positioning unit and a first Beidou positioning unit;

[0012] The first wearable device includes a second positioning module, a first camera and first AR glasses; the second positioning module includes a second GPS positioning unit and a second Beidou positioning unit;

[0013] The position of the first wearable device should be within the bird's eye view angle range of the first bird's eye camera of the first unmanned aerial vehicle.

[0014] Preferably, the method further includes:

[0015] The first unmanned aerial vehicle obtains the current positioning coordinates and the current flight height of the unmanned aerial vehicle from the first positioning module as the corresponding first body coordinates and the first body height; and generates the corresponding first bird's eye view by photographing the ground through the first bird's eye camera; and sends the first perception data composed of the first body coordinates, the first body height and the first bird's eye view to the first server.

[0016] Preferably, the method further includes:

[0017] The first wearable device obtains the current positioning coordinates and the current human body pose of the wearer through the second positioning module as the corresponding first device coordinates and the first human body pose, and generates a corresponding first image by shooting the surrounding environment through the first camera; and performs ground recognition on the first image, and labels a corresponding first ground area on the first image according to the ground recognition result; and rotates the first image according to the first human body pose and the first ground area to generate a corresponding first heads-up view; and sends the second perception data composed of the first device coordinates and the first heads-up view to the first server.

[0018] Preferably, the bird's eye view real scene map construction according to the first fuselage coordinates, the first fuselage height, the first bird's eye view and the preset high-precision map generates a corresponding first real scene map, specifically including:

[0019] According to the first fuselage coordinates, the high-precision map is projected to generate a corresponding first projection point; and the ground radius of the bird's eye view camera angle is estimated according to the first fuselage height and the internal and external parameters of the first bird's eye view camera to generate a corresponding first radius; and the first projection point is taken as the center, and the first radius is taken as the intercept radius, and the sub-high-precision map is intercepted on the high-precision map to generate a corresponding sub-high-precision map; and the first real scene map is generated by performing real scene image fusion processing on the sub-high-precision map according to the first bird's eye view.

[0020] Preferably, the AR map construction according to the first device coordinates, the first heads-up view and the first real scene map generates a corresponding overall AR map and a personal AR map, specifically including:

[0021] The target recognition processing is performed on the first heads-up view to generate a plurality of first targets; the first target includes a first target type, a first target image coordinate and a first target image; the first target type includes a person, an animal, a building, a car, a bicycle, a motorcycle and a static obstacle;

[0022] According to the first device coordinates and the internal and external parameters of the first camera, the coordinate conversion processing from image coordinates to map coordinates is performed on each first target image coordinate to obtain a corresponding first target map coordinate; and the first target image of the first target whose first target type is a person is processed by face recognition to generate a corresponding first target identity;

[0023] On the first real scene map, a corresponding AR object is created for each first target according to the first target map coordinates of the first target, and the first object coordinate attribute of each first AR object is set as the first target map coordinates of the corresponding first target, the first object type attribute of each first AR object is set as the first target type of the corresponding first target, the first object image attribute of each first AR object is set as the first target image of the corresponding first target, and when the first object type attribute of the first AR object is a person, the first object identity attribute of the current first AR object is set as the first target identity of the corresponding first target; the attributes of the first AR object include the first object coordinate attribute, the first object type attribute, the first object image attribute and the first object identity attribute;

[0024] On the first real scene map, a corresponding AR object is created for each first target according to the first target map coordinates of the first target, and the first object coordinate attribute of each first AR object is set as the first target map coordinates of the corresponding first target, the first object type attribute of each first AR object is set as the first target type of the corresponding first target, the first object image attribute of each first AR object is set as the first target image of the corresponding first target, and when the first object type attribute of the first AR object is a person, the first object identity attribute of the current first AR object is set as the first target identity of the corresponding first target; the attributes of the first AR object include the first object coordinate attribute, the first object type attribute, the first object image attribute and the first object identity attribute;

[0025] The first real scene map with completed AR object creation and setting is taken as the corresponding overall AR map, and the overall AR map is subgraph intercepted with the first device coordinates as the center point and a preset first near distance radius as the intercept radius to generate the corresponding personal AR map.

[0026] Preferably, the method further comprises:

[0027] The first wearable device receives the personal AR map sent by the first server, and calls the first AR glasses to display the personal AR map.

[0028] The second aspect of the embodiment of the application provides an electronic device, including a memory, a processor and a transceiver;

[0029] The processor is used for coupling with the memory, reading and executing instructions in the memory to realize the method steps in the first aspect;

[0030] The transceiver is coupled with the processor, and the processor controls the transceiver to perform message transmission and reception.

[0031] The third aspect of the embodiment of the present application provides a computer readable storage medium, which stores computer instructions, and when the computer instructions are executed by a computer, the computer instructions make the computer execute the method in the first aspect.

[0032] The embodiment of the present application provides a method for processing unmanned aerial vehicle and wearable device sensing data, an electronic device and a computer readable storage medium. The unmanned aerial vehicle is configured to take an aerial image to generate corresponding unmanned aerial vehicle sensing data, which is transmitted to a rear server. The wearable device is configured to take a close-range environment to generate corresponding device sensing data, which is transmitted to the rear server. The rear server creates a real scene map based on the unmanned aerial vehicle sensing data and a high-precision map, and creates an augmented reality (AR) object on the real scene map based on a target recognition result of the device sensing data and building and road semantic information of the high-precision map to obtain a corresponding overall AR map. The AR map is intercepted based on a position of each wearable device to obtain a corresponding individual AR map. The rear server displays the overall AR map to help a background dispatcher know a real scene state at the front end in the first time. The rear server pushes the individual AR map to each wearable device to help a front action person to know an environment state around in the first time and to confirm an identity of a person around. The present application improves the timeliness of the background dispatching and improves action efficiency and cooperation efficiency of the front end. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 A method for processing unmanned aerial vehicle and wearable device sensing data provided by the embodiment of the present application is shown in a schematic diagram.

[0034] Figure 2 A structure schematic diagram of an electronic device provided by the embodiment two of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0036] The background server of the command and dispatch system, i.e., the first server, can help the background dispatch personnel know the real scene state in the first time and help the front action personnel understand the surrounding environment state and confirm the identity of the surrounding personnel in the first time through the method for processing the unmanned aerial vehicle and wearable device sensing data provided by the embodiment one of the application. Figure 1 The method for processing the unmanned aerial vehicle and wearable device sensing data provided by the embodiment one of the application is shown in the figure, Figure 1 The method mainly includes the following steps:

[0037] Step 1: The first server receives the first sensing data sent by the first unmanned aerial vehicle.

[0038] The first server is connected with the first unmanned aerial vehicle through a wireless communication mode; the wireless communication mode includes 4G, 5G and WiFi communication modes; the first unmanned aerial vehicle includes a first positioning module and a first bird's eye camera; the first positioning module includes a first GPS positioning unit and a first Beidou positioning unit; the first sensing data includes first body coordinates, first body height and a first bird's eye view.

[0039] Here, the first unmanned aerial vehicle is the unmanned aerial vehicle configured by the embodiment of the application for dispatching the scene, and the first unmanned aerial vehicle is used for bird's eye image shooting of the dispatching scene and generating corresponding unmanned aerial vehicle sensing data, i.e., the first sensing data, to the rear server, i.e., the first server; the first bird's eye camera is a high-definition camera pre-installed on the first unmanned aerial vehicle, and the shooting angle of the first bird's eye camera is a bird's eye view (Bird Eyes View, BEV).

[0040] The processing steps of the first unmanned aerial vehicle of the embodiment of the application to generate the first sensing data are as follows: the first unmanned aerial vehicle obtains the current positioning coordinates and the current flight height of the unmanned aerial vehicle from the first positioning module as the corresponding first body coordinates and first body height; and generates the corresponding first bird's eye view by shooting the ground through the first bird's eye camera; and sends the first sensing data composed of the first body coordinates, the first body height and the first bird's eye view to the first server.

[0041] Step 2: Receive the second sensing data sent by the first wearable device:

[0042] The first server is connected with the first wearable device through a wireless communication mode; the wireless communication mode includes 4G, 5G and WiFi communication modes; the first wearable device includes a second positioning module, a first camera and first AR glasses; the second positioning module includes a second GPS positioning unit and a second Beidou positioning unit; the position of the first wearable device should be within the bird's eye view range of the first bird's eye camera of the first unmanned aerial vehicle; the second sensing data includes first device coordinates and a first horizon view.

[0043] Here, the first wearable device is a wearable device configured for the front action personnel at the dispatch site, the first wearable device is used to take a flat view image of the dispatch site and generate corresponding device perception data, i.e., second perception data, to the first server; the first camera is a high-definition camera pre-installed on the first wearable device; and the first AR glasses are AR glasses pre-installed on the first wearable device and supporting AR base map display.

[0044] The processing steps of the first wearable device of the embodiment of the application to generate the second perception data are specifically: the first wearable device obtains the current positioning coordinates and the current body pose of the wearer as the corresponding first device coordinates and the first body pose through the second positioning module; and generates a corresponding first image by taking a picture of the surrounding environment through the first camera; and performs ground recognition on the first image, and labels the corresponding first ground area on the first image according to the ground recognition result; and rotates the first image according to the first body pose and the first ground area to generate a corresponding first flat view; and sends the second perception data composed of the first device coordinates and the first flat view to the first server.

[0045] Here, when the front action personnel takes a picture of the surrounding environment based on the first camera of the first wearable device, it cannot be guaranteed that the body pose is completely perpendicular to the ground, that is, the generated first image is not necessarily a flat view with the ground as the baseline; therefore, after obtaining the first image, it is necessary to convert the first image to a flat view based on the corresponding real-time body pose, specifically, first, the ground area of the first image is recognized through an artificial intelligence model or the like, and then the first image is rotated according to the body pose until the ground area in the image reaches the flat view ground angle.

[0046] Step 3, according to the first body coordinates, the first body height, the first bird's eye view and the preset high-precision map, a bird's eye view real scene map is constructed to generate a corresponding first real scene map;

[0047] Specifically, it includes: step 31, according to the first body coordinates, a coordinate projection is performed on the high-precision map to generate a corresponding first projection point;

[0048] Here, the first body coordinates are obtained by the first positioning module of the first unmanned aerial vehicle, and the coordinates and the high-precision map coordinates use the same coordinate system; therefore, based on the first body coordinates, the projection point of the first unmanned aerial vehicle perpendicular to the ground on the high-precision map, i.e., the first projection point, can be found;

[0049] Step 32, according to the first body height and the internal and external parameters of the first bird's eye view camera, the ground radius of the bird's eye view camera angle is estimated to generate a corresponding first radius;

[0050] Here, the embodiment of the present application estimates the ground radius of the bird's-eye view camera view angle according to the first fuselage height and the internal and external parameters of the first bird's-eye camera using a conventional algorithm model, such as the disclosed monocular camera ranging algorithm model, YOLOv4 model, etc.; the related technical implementation can be referred to the corresponding disclosed technical materials, which will not be described one by one here; the obtained first radius is actually the shooting radius of the first unmanned aerial vehicle to the dispatch site;

[0051] Step 33, taking the first projection point as the center and the first radius as the intercept radius, a sub-high-definition map is generated by performing subgraph interception on the high-definition map;

[0052] Step 34, a first real scene map is generated by performing real scene image fusion processing on the sub-high-definition map according to the first bird's-eye view.

[0053] Here, the embodiment of the present application fuses the first bird's-eye view and the sub-high-definition map to generate a real scene map, which aims to reflect the real scene state of the dispatch site on the high-definition map. The embodiment of the present application supports multiple real scene image fusion processing methods.

[0054] One of the real scene image fusion processing methods is: a plurality of first recognition boxes are obtained by identifying the first bird's-eye view for buildings, roads and squares, wherein each first recognition box corresponds to a first recognition type, a first recognition image and a first recognition box size, the first recognition type includes buildings, roads and squares, and the first recognition box size includes ground width and ground length; and based on the first recognition box size of each first recognition box, all buildings, roads or squares on the sub-high-definition map with the same type of first recognition type are compared for similarity, and the building, road or square with the maximum similarity is taken as the first map object aligned with the current first recognition box; and taking each matched first recognition box and first map object as a reference, the first bird's-eye view and the sub-high-definition map are registered; and after registration is completed, the first recognition image of each first recognition box is taken as new map semantic information to add to the corresponding first map object on the sub-high-definition map, and the sub-high-definition map after completing the addition of new map semantic information is taken as the corresponding first real scene map. Here, the embodiment of the present application uses an image-based target recognition model to identify buildings, roads and squares in the first bird's-eye view, and such target recognition models commonly used include YOLO model, R-CNN model, fast R-CNN model, etc.

[0055] Another real scene image fusion processing mode needs to set a plurality of obvious calibration reference objects in the shooting range of the first unmanned aerial vehicle in advance to ensure that all the calibration reference objects are included in the first bird's eye view, and it is confirmed that the map coordinates and the size of each calibration reference object are known. When performing real scene image fusion processing, the first bird's eye view is subjected to calibration reference object identification to obtain a plurality of second identification boxes, wherein each second identification box corresponds to a second identification type, a second identification image and a second identification box size, the second identification type includes calibration reference objects and non-calibration reference objects, and the second identification box size includes ground width and ground length. The second identification box with the second identification type of the calibration reference object is marked as a calibration reference object identification box. The center points of all the calibration reference object identification boxes in the first bird's eye view are sequentially connected to obtain a corresponding polygon image. The map coordinates corresponding to all the calibration reference object identification boxes on the sub-high-precision map are sequentially connected to obtain a corresponding polygon map, which is consistent in shape with the above-mentioned polygon image. The polygon image and the polygon map are overlapped and fused to obtain a corresponding first real scene map output. Here, the embodiment of the application uses an image-based target identification model to identify the calibration reference objects in the first bird's eye view. Commonly used target identification models include YOLO model, R-CNN model, fast R-CNN model, etc.

[0056] Step 4, AR map construction is performed according to the first device coordinates, the first flat view and the first real scene map to generate a corresponding overall AR map and a personal AR map.

[0057] Specifically, step 41, target identification processing is performed on the first flat view to generate a plurality of first targets.

[0058] The first target includes a first target type, a first target image coordinate and a first target image. The first target type includes people, animals, buildings, cars, bicycles, motorcycles and static obstacles.

[0059] Here, the embodiment of the application uses an image-based target identification model to identify the targets in the first flat view. Commonly used target identification models include YOLO model, R-CNN model, fast R-CNN model, etc.

[0060] Step 42, coordinate conversion processing is performed on each first target image coordinate from image coordinates to map coordinates according to the first device coordinates and the internal and external parameters of the first camera to obtain corresponding first target map coordinates. Face recognition processing is performed on the first target image of the first target with the first target type of people to generate corresponding first target identities.

[0061] Here, the first server of the embodiment of the application can obtain the identity information and the frontal image of each front action person locally, and can also obtain more personnel identity information by connecting a third-party identity verification platform or an identity information library; when the first server of the embodiment of the application performs face recognition processing on the first target image of the first target of the first target type of person: first, face feature extraction is performed on the first target image to obtain the corresponding first target feature map; and whether a matching person is found is confirmed by comparing the first target feature map with the feature comparison results of each frontal image saved locally; if it is confirmed that a matching person is found, the identity information and the frontal image of the person are taken as the corresponding first target identity; if it is confirmed that a matching person is not found, the first target image is sent to the third-party identity verification platform or the identity information library for comparison, and the personnel identity information returned by the third-party identity verification platform or the identity information library is taken as the corresponding first target identity; it should be noted that the third-party identity verification platform or the identity information library mentioned here can be multiple;

[0062] Step 43, on the first real scene map, according to the first target map coordinates, create corresponding AR objects for each first target, denoted as corresponding first AR objects; and set the first object coordinate attribute of each first AR object to the corresponding first target map coordinates of the corresponding first target; and set the first object type attribute of each first AR object to the first target type of the corresponding first target; and set the first object image attribute of each first AR object to the first target image of the corresponding first target; and when the first object type attribute of the first AR object is a person, set the first object identity attribute of the current first AR object to the corresponding first target identity of the corresponding first target;

[0063] Among them, the attributes of the first AR object include the first object coordinate attribute, the first object type attribute, the first object image attribute and the first object identity attribute;

[0064] Here, the embodiment of the application creates corresponding virtual AR objects, i.e. first AR objects, for people, animals, buildings, cars, bicycles, motorcycles and static obstacles appearing in the first heads-up view on the first real scene map based on augmented reality technology;

[0065] Step 44, on the first real scene map, according to the building and road semantic information of the high-precision map, corresponding AR object creation is generated for each building and road which has not created AR object to generate corresponding second AR object; and the second object coordinate attribute of each second AR object is set according to the building and road coordinates of the building and road semantic information of the high-precision map; and the second object type attribute of each second AR object is set according to the building type and road type of the building and road semantic information of the high-precision map; and the second object identity attribute of each second AR object is set according to the building name and road name of the building and road semantic information of the high-precision map;

[0066] Among them, the attributes of the second AR object include the second object coordinate attribute, the second object type attribute and the second object identity attribute;

[0067] Here, the embodiment of the application creates corresponding virtual AR objects, i.e. second AR objects, for the remaining buildings and all roads on the high-precision map on the first real scene map based on augmented reality technology;

[0068] Step 45, the first real scene map after completing AR object creation and setting is taken as a corresponding overall AR map; and the personal AR map is generated by taking the first device coordinate as the center point and taking the preset first close distance radius as the intercept radius to intercept the overall AR map.

[0069] Here, the overall AR map is a panoramic AR map constructed based on the first real scene map generated by the first unmanned aerial vehicle image, which helps the background dispatch personnel to know the overall real scene state in the first time; the first close distance radius is a pre-set distance radius parameter, which is usually smaller than the first radius estimated in the foregoing; the personal AR map is a sub-map intercepted from the panoramic AR map based on the actual position of a front moving personnel.

[0070] Step 5, the overall AR map is displayed; and the personal AR map is sent to the first wearable device.

[0071] Here, the first server of the embodiment of the application displays the overall AR map in the command and dispatch center through the external display device, and at the same time, the personal AR map is sent to the first wearable device. When the first wearable device in the front end receives the personal AR map sent by the first server, the first AR glasses are called to display the personal AR map.

[0072] To sum up, the first server of the embodiment of the application can obtain the corresponding overall AR map and personal AR map by processing the sensing data of the unmanned aerial vehicle and the wearable device through the above steps 1-5, and can help the background dispatch personnel to know the real scene state in the front in the first time by displaying the overall AR map, and can help the front action personnel to understand the surrounding environment state in the first time by pushing the personal AR map to each wearable device, and can help to confirm the identity of the surrounding personnel.

[0073] It should be noted that the above steps 1-5 give a processing flow based on the sensing data of one unmanned aerial vehicle and one wearable device, and similar to the above steps 1-5, a processing flow for the sensing data of one unmanned aerial vehicle and multiple wearable devices can be further expanded, specifically as follows:

[0074] Step 101, the first server receives the third sensing data sent by the first unmanned aerial vehicle;

[0075] Among them, the third sensing data includes the second body coordinate, the second body height and the second bird's eye view;

[0076] Here, the current step 101 is similar to the aforementioned step 1;

[0077] Step 102, receiving the fourth sensing data sent by multiple first wearable devices:

[0078] Among them, the fourth sensing data includes the second device coordinate and the second flat view;

[0079] Here, the current step 102 is similar to the aforementioned step 2;

[0080] Step 103, according to the second body coordinate, the second body height, the second bird's eye view and the preset high-precision map, the bird's eye view real scene map is constructed to generate the corresponding second real scene map;

[0081] Here, the current step 102 is similar to the aforementioned step 3;

[0082] Step 104, according to the second device coordinate, the second flat view and the second real scene map of each fourth sensing data, the AR map is constructed to generate the corresponding first overall AR map and multiple first personal AR maps;

[0083] Here, the current step 104 is similar to the aforementioned step 4, the difference is that the aforementioned step 4 only constructs the AR map according to the first device coordinate, the first flat view and the first real scene map of one second sensing data, while the current step 104 constructs the AR map based on the second device coordinate, the second flat view and the second real scene map of multiple fourth sensing data;

[0084] Specifically comprising: step 1041, traversing each second perception data, taking the currently traversed second perception data as the current perception data, taking the first device coordinate and the first plan view of the current perception data as the corresponding current device coordinate and current plan view, and taking the internal and external parameters of the first camera corresponding to the current perception data as the corresponding current camera internal and external parameters; and performing target recognition processing on the current plan view to generate a plurality of second targets, wherein the second target includes a second target type, a second target image coordinate and a second target image, the second target type includes a person, an animal, a building, a car, a bicycle, a motorcycle and a static obstacle; and performing coordinate conversion processing from image coordinate to map coordinate on each second target image coordinate according to the current device coordinate and the current camera internal and external parameters to obtain the corresponding second target map coordinate; and performing face recognition processing on the second target image of the second target whose second target type is a person to generate the corresponding second target identity; and on the second real scene map, creating the corresponding AR object for each second target according to the second target map coordinate, denoted as the corresponding third AR object, wherein the attribute of the third AR object includes a third object coordinate attribute, a third object type attribute, a third object image attribute and a third object identity attribute; and setting the third object coordinate attribute of each third AR object as the second target map coordinate corresponding to the corresponding second target; and setting the third object type attribute of each third AR object as the second target type of the corresponding second target; and setting the third object image attribute of each third AR object as the second target image of the corresponding second target; and when the third object type attribute of the third AR object is a person, setting the third object identity attribute of the current third AR object as the second target identity corresponding to the corresponding second target;

[0085] Here, the current step 1041 is actually to repeatedly execute the aforementioned steps 41-43, so as to create the corresponding virtual AR object, i.e., the third AR object, for all the persons, animals, buildings, cars, bicycles, motorcycles and static obstacles appearing in the second plan view returned by the second wearable device on the second real scene map based on the augmented reality technology;

[0086] Step 1042, on the second real scene map, according to the building and road semantic information of the high-precision map, corresponding fourth AR objects are generated for the respective buildings and roads which have not created AR objects; and the fourth object coordinate attribute of each fourth AR object is set according to the building and road coordinates of the building and road semantic information of the high-precision map; and the fourth object type attribute of each fourth AR object is set according to the building type and road type of the building and road semantic information of the high-precision map; and the fourth object identity attribute of each fourth AR object is set according to the building name and road name of the building and road semantic information of the high-precision map; wherein the attributes of the fourth AR object include the fourth object coordinate attribute, the fourth object type attribute and the fourth object identity attribute;

[0087] Here, the current step 1042 is actually similar to the aforementioned step 44, that is, based on augmented reality technology, corresponding virtual AR objects, i.e. fourth AR objects, are created for the remaining buildings and all roads on the high-precision map on the second real scene map;

[0088] Step 1043, the second real scene map after completing AR object creation and setting is taken as a corresponding first overall AR map; and the first overall AR map is subgraph intercepted to generate a corresponding first personal AR map, with the second device coordinates of each first wearable device as the center point and the preset first close distance radius as the intercept radius.

[0089] Here, the current step 1043 is actually similar to the aforementioned step 44, except that when generating a personal AR map, the aforementioned step 44 only generates one, while the current step 1043 generates multiple.

[0090] Step 105, the first overall AR map is displayed; and each first personal AR map is sent to the corresponding first wearable device.

[0091] Here, the current step 105 is actually similar to the aforementioned step 5, and the first server of the embodiment of the application displays the first overall AR map in the command center through the external display device, and at the same time sends each first personal AR map to the corresponding first wearable device. When each first wearable device at the front end receives the first personal AR map sent by the first server, the first AR glasses are called to display the first AR map.

[0092] Figure 2 A structural schematic diagram of an electronic device provided by the second embodiment of the application. The electronic device can be the terminal device or server mentioned above, or a terminal device or server connected to the terminal device or server mentioned above and realizing the method of the embodiment of the application. For example, Figure 2As shown, the electronic device can include a processor 301 (e.g., a CPU), a memory 302, and a transceiver 303. The transceiver 303 is coupled to the processor 301, and the processor 301 controls the transceiving action of the transceiver 303. The memory 302 can store various instructions for completing various processing functions and implementing the processing steps described in the foregoing method embodiments. Preferably, the electronic device related to the embodiments of the present application further includes a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize the communication connection between elements. The communication port 306 described above is used for the connection communication between the electronic device and other peripherals.

[0093] In Figure 2 The system bus 305 mentioned in the foregoing can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus. The communication interface is used to realize the communication between the database access device and other devices (e.g., a client, a read-write library, and a read-only library). The memory can include a Random Access Memory (RAM), and can also include a Non-Volatile Memory, such as at least one disk memory.

[0094] The processor described above can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), a Graphics Processing Unit (GPU), etc.; can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0095] It should be noted that the embodiments of the present application also provide a computer readable storage medium, which stores instructions, and when the instructions are run on a computer, the computer executes the method and processing procedure provided in the foregoing embodiments.

[0096] The embodiment of the present application also provides a chip for running instructions, which is used for executing the processing steps described in the foregoing method embodiment.

[0097] The embodiment of the present application provides a method for processing unmanned aerial vehicle and wearable device sensing data, an electronic device and a computer readable storage medium. The unmanned aerial vehicle is configured to take an aerial image to generate corresponding unmanned aerial vehicle sensing data, which is transmitted to a rear server. The wearable device is configured to take a close-range environment to generate corresponding device sensing data, which is transmitted to the rear server. The rear server creates a real scene map based on the unmanned aerial vehicle sensing data and a high-precision map, and creates an augmented reality (AR) object on the real scene map based on a target recognition result of the device sensing data and building and road semantic information of the high-precision map to obtain a corresponding overall AR map. The AR map is intercepted based on a position of each wearable device to obtain a corresponding individual AR map. The rear server displays the overall AR map to help a background dispatcher know a real scene state at a front end in a first time. The rear server pushes the individual AR map to each wearable device to help a front action person know an environment state around in a first time and confirm an identity of a person around. The present application improves timeliness of background dispatching and action efficiency and cooperation efficiency of the front end.

[0098] Those skilled in the art will further appreciate that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, various aspects of each example have been described generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Skilled persons can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0099] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0100] The above detailed description of the specific embodiments of the present application has been given to understand the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for processing data sensed by a drone and a wearable device, the method comprising: The method comprises: The first server receives the first perception data sent by the first unmanned aerial vehicle; the first perception data comprises first body coordinates, first body height and first bird's-eye view; The second perception data sent by the first wearable device is received: the second perception data comprises first device coordinates and first horizon view; The first server is connected with the first unmanned aerial vehicle and the first wearable device through wireless communication mode; the wireless communication mode comprises 4G, 5G and WiFi communication mode; The first unmanned aerial vehicle comprises a first positioning module and a first bird's-eye camera; the first positioning module comprises a first GPS positioning unit and a first Beidou positioning unit; The first wearable device comprises a second positioning module, a first camera and first AR glasses; the second positioning module comprises a second GPS positioning unit and a second Beidou positioning unit; The position of the first wearable device is within the bird's-eye view angle range of the first bird's-eye camera of the first unmanned aerial vehicle; According to the first body coordinates, the first body height, the first bird's-eye view and the preset high-precision map, the bird's-eye real scene map construction is generated to generate the corresponding first real scene map; specifically comprising: According to the first body coordinates, the first body height, the first bird's-eye view and the preset high-precision map, the bird's-eye real scene map construction is generated to generate the corresponding first real scene map; specifically comprising: According to the first device coordinates, the first horizon view and the first real scene map, the AR map construction is generated to generate the corresponding overall AR map and personal AR map; The overall AR map is displayed; and the personal AR map is sent to the first wearable device. 2.The method for processing data perceived by a UAV and a wearable device according to claim 1, wherein, The method further comprises: The first unmanned aerial vehicle obtains the current positioning coordinates and the current flight height of the unmanned aerial vehicle from the first positioning module as the corresponding first body coordinates and first body height; and the ground is photographed by the first bird's-eye camera to generate the corresponding first bird's-eye view; and the first perception data composed of the first body coordinates, the first body height and the first bird's-eye view is sent to the first server. 3.The method for processing data perceived by a UAV and a wearable device according to claim 1, wherein, The method further comprises: The first wearable device obtains the current positioning coordinates and the current human body pose of the wearer through the second positioning module as the first device coordinates and the first human body pose, respectively, and generates a first image by capturing the surrounding environment through the first camera. The first image is ground identified, and the first ground area is labeled on the first image according to the ground identification result. The first image is rotated according to the first human body pose and the first ground area to generate a first heads-up view. The second perception data composed of the first device coordinates and the first heads-up view is sent to the first server. 4.The method for processing data perceived by a UAV and a wearable device according to claim 1, wherein, The first device coordinates, the first heads-up view, and the first real scene map are used to construct an AR map to generate a corresponding overall AR map and a personal AR map, which specifically includes: target identification processing of the first heads-up view to generate a plurality of first targets; the first target includes a first target type, a first target image coordinate, and a first target image; the first target type includes a person, an animal, a building, a car, a bicycle, a motorcycle, and a static obstacle; According to the internal and external parameters of the first device coordinates and the first camera, the coordinate conversion processing from image coordinates to map coordinates is performed on each first target image coordinate to obtain the corresponding first target map coordinate; and the first target image of the first target whose first target type is a person is processed by face recognition to generate a corresponding first target identity. On the first real scene map, according to the first target map coordinate, each first target creates a corresponding AR object, denoted as a corresponding first AR object; the first object coordinate attribute of each first AR object is set to the first target map coordinate corresponding to the corresponding first target; the first object type attribute of each first AR object is set to the first target type of the corresponding first target; the first object image attribute of each first AR object is set to the first target image of the corresponding first target; when the first object type attribute of the first AR object is a person, the first object identity attribute of the current first AR object is set to the first target identity corresponding to the corresponding first target; the attributes of the first AR object include the first object coordinate attribute, the first object type attribute, the first object image attribute, and the first object identity attribute; On the first real scene map, according to the building and road semantic information of the high-precision map, the corresponding AR object creation is generated for each building and road which has not created the AR object, to generate the corresponding second AR object; and according to the building and road coordinates of the building and road semantic information of the high-precision map, the second object coordinate attribute of each second AR object is set; and according to the building type and road type of the building and road semantic information of the high-precision map, the second object type attribute of each second AR object is set; and according to the building name and road name of the building and road semantic information of the high-precision map, the second object identity attribute of each second AR object is set; the attributes of the second AR object include the second object coordinate attribute, the second object type attribute and the second object identity attribute; The first real scene map which completes the AR object creation and setting is taken as the corresponding whole AR map; and the personal AR map is generated by taking the first device coordinate as the center point and taking the preset first near distance radius as the intercepting radius to intercept the subgraph of the whole AR map. 5.The method for processing data perceived by a UAV and a wearable device according to claim 1, wherein, The method further comprises: When the first wearing device receives the personal AR map sent by the first server, the first AR glasses are called to display the personal AR map.

6. An electronic device, comprising: Comprise: Memory, processor and transceiver; The processor is used for coupling with the memory, reading and executing the instructions in the memory, so as to realize the method steps of any one of claims 1-5; The transceiver is coupled with the processor, and the transceiver is controlled by the processor to perform message transceiving.

7. A computer readable storage medium characterized by The computer readable storage medium stores computer instructions, when the computer instructions are executed by the computer, the computer instructions make the computer execute the instructions of the method of any one of claims 1-5.

Images