Multi-terminal adaptive data processing method, system, device and storage medium

Abstract

The present invention belongs to the technical field of automatic data collection and processing, and in particular, relates to a multi-terminal adaptive data processing method, system, device and storage medium; the present invention uses information interaction between multiple controllable aircraft terminals and a control terminal, and uses cameras and far-infrared thermal imagers carried by each controllable aircraft terminal to cooperate with each other to lock and track a target, and locate the exit position information of each building in a 3D map. When the tracking target enters a building and loses the tracking object, other controllable aircraft terminals can fly to other exits of the building in time for monitoring, and use the far-infrared thermal imager to cooperate with the video stream obtained by the camera to re-identify and locate the tracking target, so as to prevent the tracking target from escaping by passing through a complex building.

Description

Technical Field

[0001] The present invention belongs to the technical field of automatic data collection and processing, and in particular to a multi-terminal adaptive data processing method, system, device and storage medium. Background Art

[0002] The Chinese patent with the authorization announcement number CN112532870B discloses a multi-terminal adaptive data processing system, method and computer-readable storage medium. The data processing system includes multiple data acquisition terminals, each of which includes an image memory, an edge processor and multiple micro image sensors with different resolutions; the edge processor is also equipped with a broadcast antenna, through which the edge processor broadcasts the result of its own processing of the image data to the edge processors of other data acquisition terminals, and receives the image data processing results broadcasted by the broadcast antennas configured by the edge processors of other data acquisition terminals; based on the image data processing results, the edge processor controls the enabling state of its own micro image sensors with different resolutions. It can be applied to the real-time tracking of specific target persons on the scene, such as crime scene pursuit or post-crime tracking; it can also be applied to the identification of specific target buildings in a specific area, such as illegal building inspections, etc.; of course, the present invention can also be used for character summary reports at large gatherings, such as real-time reports of champions at large sports events, etc.

[0003] However, the above patent solutions still have shortcomings in tracking crime scenes. For example, the movement trajectory of the suspect is complex. During the tracking process, he will enter and exit the surrounding buildings, resulting in the camera on the drone being unable to obtain the position of the tracked object in real time. The tracked object is prone to change clothes in the tracking blind spot, and the process of tracking by camera is prone to losing the tracking target in the blind spot of vision. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a multi-terminal adaptive data processing method, system, device and storage medium to solve the problem that the tracking target is easily lost in the blind spot of the field of vision in the prior art for crime scene tracking.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A multi-terminal adaptive data processing method,

[0007] Adaptive data collection is achieved based on n controllable aircraft terminals, each controllable aircraft is equipped with a camera and a far-infrared thermal imager, and a control terminal that is communicatively connected to the n controllable aircraft terminals, the control terminal stores 3D map information within the range of the tracking target, the 3D map information includes the location information of each exit of the building within the range, and the control terminal has an image display module, the image display module is used to display the video stream obtained by the camera of each controllable aircraft terminal and the thermal imaging video screen stream obtained by the far-infrared thermal imager;

[0008] The data processing method comprises the following steps:

[0009] S1: The control terminal sends 3D map information within the location range of the tracking target to n controllable aircraft terminals;

[0010] S2: Each controllable aircraft terminal searches for a flight path according to the 3D map information plan. The camera carried by each controllable aircraft obtains and sends a video stream of the search path to the control terminal in real time. When the control terminal obtains image information with a high similarity to the image information of the tracking target in the video stream obtained by one of the controllable aircraft terminals, the controllable aircraft terminal is controlled as the first tracking aircraft to lock and follow the tracking target, and the far-infrared thermal imager carried by the controllable aircraft terminal is controlled to turn on to obtain the far-infrared thermal imaging video of the tracking target in real time.

[0011] S3: Control other controllable aircraft terminals to fly around the first tracking aircraft with the first tracking aircraft as the center;

[0012] S4: If the first tracking aircraft loses the tracking target, it is determined that the tracking target has entered one of the buildings in the 3D map according to the location information of the lost tracking target and the 3D map information. The first tracking aircraft stays and moves toward the location where the tracking target is lost. According to the number of exits of the building in the 3D map being m, the other m-1 controllable aircraft terminals nearby are controlled to move to other exits of the building in the 3D map, and at the same time, the m-1 controllable aircraft terminals turn on the far-infrared thermal imagers and use their cameras and far-infrared thermal imagers to move toward other exits of the building.

[0013] S5: Each controllable aircraft terminal obtains the far-infrared thermal imaging video stream of the newly appeared target at each exit of the building, and compares it with the far-infrared thermal imaging information of the previously obtained tracking target. If the similarity is greater than a preset threshold, the corresponding controllable aircraft terminal acts as a new first tracking aircraft to lock and follow the target;

[0014] S6: Control other controllable aircraft terminals to fly around the first tracking aircraft with the first tracking aircraft as the center.

[0015] Furthermore, in the above-mentioned multi-terminal adaptive data processing method, the data processing method also includes S7: the camera of the new first tracking aircraft obtains the video stream of the target, and compares it with the image of the previous tracking target. If the similarity is less than a preset threshold, the image of the tracking target is updated to the latest acquired tracking target image information, and the first tracking aircraft switches to use the updated tracking target image instead of thermal imaging as the tracking object.

[0016] Furthermore, in the above multi-terminal adaptive data processing method, the first tracking aircraft tracks the target locked by the camera based on a PID control algorithm.

[0017] Furthermore, in the above-mentioned multi-terminal adaptive data processing method, in S3 or S6, when the other controllable aircraft terminals follow the first tracking aircraft around the first tracking aircraft, the flight altitude of the other controllable aircraft terminals is higher than that of the first tracking aircraft, and they lock and track the targets at a long distance respectively.

[0018] Furthermore, in the above multi-terminal adaptive data processing method, in S3 or S6, if the first tracking aircraft fails, another controllable aircraft terminal closest to the tracking target is switched to the first tracking aircraft to perform the tracking task of the original first tracking aircraft.

[0019] The present invention also relates to a multi-terminal adaptive data processing system, comprising:

[0020] The information sending and receiving module of the control terminal is used to send 3D map information within the location range of the tracking target to n controllable aircraft terminals, and receive video stream information collected by the camera and far-infrared thermal imager of each controllable aircraft terminal;

[0021] The flight control module of each controllable aircraft terminal is used to track the flight according to the 3D map information and the flight instructions sent by the control terminal according to the tracking algorithm;

[0022] The storage module of the control terminal is used to store 3D map information, as well as tracking target camera images and far-infrared thermal imaging image information in the video stream acquired by the controllable aircraft terminal;

[0023] The image comparison and recognition module of the control terminal is used to compare the tracking target camera image and far-infrared thermal imaging image information in the video stream obtained by the controllable aircraft terminal with the tracking target camera image and far-infrared thermal imaging image information in the stored video stream to identify the tracking target.

[0024] The present invention also relates to a multi-terminal adaptive data processing device, comprising a memory, a processor and a multi-terminal adaptive data processing program stored in the memory and executable on the processor, wherein the multi-terminal adaptive data processing program is configured to implement the steps of the multi-terminal adaptive data processing method described above.

[0025] The present invention also relates to a storage medium, on which a multi-terminal adaptive data processing program is stored, and when the multi-terminal adaptive data processing program is executed by a processor, the steps of the multi-terminal adaptive data processing method are implemented.

[0026] The beneficial effects of the present invention are as follows: the present invention uses information interaction between multiple controllable aircraft terminals and a control terminal, and uses the cameras and far-infrared thermal imagers carried by each controllable aircraft terminal to work together to lock and track the target, and locate the exit position information of each building in the 3D map. When the tracking target enters the building and loses the tracking object, other controllable aircraft terminals can fly to other exits of the building in time for monitoring, and use the far-infrared thermal imager to cooperate with the video stream obtained by the camera to re-identify and locate the tracking target, so as to prevent the tracking target from escaping by passing through complex buildings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A flowchart of a multi-terminal adaptive data processing method according to a specific embodiment of the present invention; DETAILED DESCRIPTION

[0028] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0029] Please refer to Figure 1 The specific implementation of the present invention relates to a multi-terminal adaptive data processing method.

[0030] Adaptive data collection is achieved based on n controllable aircraft terminals, each controllable aircraft is equipped with a camera and a far-infrared thermal imager, and a control terminal that is communicatively connected to the n controllable aircraft terminals, the control terminal stores 3D map information within the range of the tracking target, the 3D map information includes the location information of each exit of the building within the range, and the control terminal has an image display module, the image display module is used to display the video stream obtained by the camera of each controllable aircraft terminal and the thermal imaging video screen stream obtained by the far-infrared thermal imager;

[0031] It should be noted that the 3D map information mainly marks the locations of the exits of the buildings. The 3D map needs to be updated regularly to ensure that the exit locations of the buildings in each area in the 3D map are consistent with the exit locations of the actual buildings.

[0032] The data processing method comprises the following steps:

[0033] S1: The control terminal sends 3D map information within the location range of the tracking target to n controllable aircraft terminals;

[0034] In this step, the photo information of the tracking target such as the suspect and the location of the crime scene are sent to the control terminal when reporting the case. The control terminal extracts the image features in the photo of the suspect as the comparison object; the control terminal determines the initial search range according to the location of the crime scene, and sends the 3D map information within the location range to n controllable aircraft terminals. The number of controllable aircraft terminals can be flexibly selected according to the complexity of the terrain in the area. For example, for terrain with dense buildings and complex exit locations, the number of controllable aircraft terminals sent out for search is relatively large. For terrain with sparse buildings, the number of controllable aircraft terminals sent out for search is relatively small.

[0035] S2: Each controllable aircraft terminal searches for a flight path according to the 3D map information plan. The camera carried by each controllable aircraft obtains and sends a video stream of the search path to the control terminal in real time. When the control terminal obtains image information with a high similarity to the image information of the tracking target in the video stream obtained by one of the controllable aircraft terminals, the controllable aircraft terminal is controlled as the first tracking aircraft to lock and follow the tracking target, and the far-infrared thermal imager carried by the controllable aircraft terminal is controlled to turn on to obtain the far-infrared thermal imaging video of the tracking target in real time.

[0036] In this step, each controllable aircraft terminal plans and searches for the flight path according to the 3D map information, which can be achieved by the existing flight control system, including obstacle avoidance through the lidar system, allocating planned paths for separate searches, etc. The specific flight control algorithm is not an improvement of the present invention, so it will not be elaborated.

[0037] In this step, the video stream of the search path is obtained through the camera of each controllable aircraft. A human-shaped object can be captured as the search object, and the image in the video stream of the search object obtained is compared with the photo of the suspect through the existing deep learning algorithm. In the image comparison during movement, motion estimation and compensation technology can be considered to reduce the image difference caused by movement; in scenes with large illumination changes, illumination invariance feature extraction technology can be used to improve the robustness of the algorithm; after obtaining the target object with high similarity through image comparison, it can be further confirmed by human eye recognition to determine that the tracking target is the suspect. At this time, the controllable aircraft terminal, as the first tracking aircraft, locks and follows the tracking target, and controls the far-infrared thermal imager carried by the controllable aircraft terminal to turn on, and obtains the far-infrared thermal imaging video of the tracking target in real time; the control terminal records the far-infrared thermal imaging video information of the tracking target, including the thermal imaging image information of each frame in the video stream.

[0038] S3: Control other controllable aircraft terminals to fly around the first tracking aircraft with the first tracking aircraft as the center;

[0039] In this step, since the tracking target is locked, other controllable aircraft terminals performing search tasks fly around the first tracking aircraft with the first tracking aircraft as the center. On the one hand, they can assist in following the movement of the suspect, and on the other hand, when the suspect enters the building (visual blind spot), they can quickly move to other exits of the building and wait, so as to prevent the suspect from escaping the tracking from the visual blind spot.

[0040] S4: If the first tracking aircraft loses the tracking target, it is determined that the tracking target has entered one of the buildings in the 3D map according to the location information of the lost tracking target and the 3D map information. The first tracking aircraft stays and moves toward the location where the tracking target is lost. According to the number of exits of the building in the 3D map being m, the other m-1 controllable aircraft terminals nearby are controlled to move to other exits of the building in the 3D map, and at the same time, the m-1 controllable aircraft terminals turn on the far-infrared thermal imagers and use their cameras and far-infrared thermal imagers to move toward other exits of the building.

[0041] In this step, for example, if the exit of the building where the tracking target is located is m=4, and the number of controllable aircraft terminals previously dispatched is n=8, then the first tracking aircraft previously stays at one of the exits, and the three controllable aircraft terminals flying around it move to the other three exits of the building respectively, and the remaining controllable aircraft terminals continue to fly around and are in a standby state;

[0042] S5: Each controllable aircraft terminal obtains the far-infrared thermal imaging video stream of the newly appeared target at each exit of the building, and compares it with the far-infrared thermal imaging information of the previously obtained tracking target. If the similarity is greater than a preset threshold, the corresponding controllable aircraft terminal acts as a new first tracking aircraft to lock and follow the target;

[0043] In this step, considering that the target criminal suspect may change his clothes in the building (in the visual blind spot), the appearance image of the target criminal suspect is very different from the original appearance image. If the original tracking image is still used for tracking, the tracking target will be lost, and the criminal suspect will escape. At this time, the far-infrared thermal imaging video stream information of the human-shaped object at each exit of the building is obtained by using the far-infrared thermal imager of each controllable aircraft terminal, and the thermal imaging information in the far-infrared thermal imaging video stream information obtained before is compared frame by frame. It is judged whether it is the criminal suspect by the similarity. If so, the corresponding controllable aircraft terminal is used as the new first tracking aircraft to lock and track the target;

[0044] In this step, the body temperature radiation of different people is different. Since far-infrared thermal imaging captures the radiation temperature distribution difference of the target object, even if the target object changes clothes, its body temperature distribution will not change. The present invention uses this property to capture the far-infrared thermal imaging image of the human-shaped object at the exit of the target building and compare it with the pre-stored thermal imaging image of the target object, so as to re-determine the locked target tracking object and avoid the target tracking object escaping due to the tracking object changing clothes.

[0045] S6: Control other controllable aircraft terminals to fly around the first tracking aircraft with the first tracking aircraft as the center.

[0046] As a preferred embodiment, the data processing method further includes S7: the camera of the new first tracking aircraft obtains the video stream of the target and compares it with the image of the previous tracking target. If the similarity is less than a preset threshold, the image of the tracking target is updated to the latest acquired tracking target image information, and the first tracking aircraft switches to use the updated tracking target image instead of thermal imaging as the tracking object.

[0047] In this step, when capturing the far-infrared thermal imaging of the tracking object as the tracking target, the far-infrared thermal imaging itself has certain defects. Since the far-infrared thermal imaging senses the heat radiated by the object, for example, when the surrounding human-shaped objects overlap, there is a risk of erroneous locking of the target object. After the tracking target is locked through the far-infrared thermal imaging, the image information taken after the suspect changes his clothes is updated in time, and the first tracking aircraft switches to use the updated image of the tracking target instead of the thermal imaging as the tracking object, so as to ensure the stability of the tracking.

[0048] As a preferred embodiment, the first tracking aircraft tracks the target locked by the camera based on a PID control algorithm.

[0049] The PID algorithm control algorithm achieves precise control of the UAV tracking target through the synergistic effect of the three links of proportion, integration and differentiation. Proportional control enables the UAV to quickly respond to changes in the target position, integral control eliminates steady-state errors, and differential control improves the stability and accuracy of tracking. These characteristics make the PID control algorithm widely used in the field of UAV tracking.

[0050] As a preferred embodiment, in S3 or S6, when the other controllable aircraft terminals follow the first tracking aircraft around the first tracking aircraft, the flight altitude of the other controllable aircraft terminals is higher than that of the first tracking aircraft, and they lock and track the targets at a long distance respectively.

[0051] In this step, other controllable aircraft terminals are mainly in a standby state following the flight of the first tracking aircraft, so that after the suspect enters the building and loses the tracking target, they can fly to other exit locations of the building in time. The altitude of other controllable aircraft terminals is higher than that of the first tracking aircraft. On the one hand, they fly at a safe altitude to avoid human damage, and on the other hand, they can reduce the number and complexity of obstacles and make the flight smoother.

[0052] As a preferred implementation, in S3 or S6, if the first tracking aircraft fails, another controllable aircraft terminal closest to the tracking target is switched to the first tracking aircraft to perform the tracking task originally performed by the first tracking aircraft.

[0053] In this step, if the first tracking aircraft is considered to be damaged or failed in obstacle avoidance, other controllable aircraft terminals can replace the first tracking aircraft in time to perform the tracking task.

[0054] The present invention also relates to a multi-terminal adaptive data processing system, comprising:

[0055] The information sending and receiving module of the control terminal is used to send 3D map information within the location range of the tracking target to n controllable aircraft terminals, and receive video stream information collected by the camera and far-infrared thermal imager of each controllable aircraft terminal;

[0056] The flight control module of each controllable aircraft terminal is used to track the flight according to the 3D map information and the flight instructions sent by the control terminal according to the tracking algorithm;

[0057] The storage module of the control terminal is used to store 3D map information, as well as tracking target camera images and far-infrared thermal imaging image information in the video stream acquired by the controllable aircraft terminal;

[0058] The image comparison and recognition module of the control terminal is used to compare the tracking target camera image and far-infrared thermal imaging image information in the video stream obtained by the controllable aircraft terminal with the tracking target camera image and far-infrared thermal imaging image information in the stored video stream to identify the tracking target.

[0059] The present invention also relates to a multi-terminal adaptive data processing device, comprising a memory, a processor and a multi-terminal adaptive data processing program stored in the memory and executable on the processor, wherein the multi-terminal adaptive data processing program is configured to implement the steps of the multi-terminal adaptive data processing method described above.

[0060] The present invention also relates to a storage medium, on which a multi-terminal adaptive data processing program is stored, and when the multi-terminal adaptive data processing program is executed by a processor, the steps of the multi-terminal adaptive data processing method are implemented.

[0061] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multi-terminal adaptive data processing method, characterized in that: Adaptive data collection is achieved based on n controllable aircraft terminals, each controllable aircraft is equipped with a camera and a far-infrared thermal imager, and a control terminal that is communicatively connected to the n controllable aircraft terminals, the control terminal stores 3D map information within the range of the tracking target, the 3D map information includes the location information of each exit of the building within the range, and the control terminal has an image display module, the image display module is used to display the video stream obtained by the camera of each controllable aircraft terminal and the thermal imaging video screen stream obtained by the far-infrared thermal imager; The data processing method comprises the following steps: S1: The control terminal sends 3D map information within the location range of the tracking target to n controllable aircraft terminals; S2: Each controllable aircraft terminal searches for a flight path according to the 3D map information plan. The camera carried by each controllable aircraft obtains and sends a video stream of the search path to the control terminal in real time. When the control terminal obtains image information with a high similarity to the image information of the tracking target in the video stream obtained by one of the controllable aircraft terminals, the controllable aircraft terminal is controlled as the first tracking aircraft to lock and follow the tracking target, and the far-infrared thermal imager carried by the controllable aircraft terminal is controlled to turn on to obtain the far-infrared thermal imaging video of the tracking target in real time. S3: Control other controllable aircraft terminals to fly around the first tracking aircraft with the first tracking aircraft as the center; S4: If the first tracking aircraft loses the tracking target, it is determined that the tracking target has entered one of the buildings in the 3D map according to the location information of the lost tracking target and the 3D map information. The first tracking aircraft stays and moves toward the location where the tracking target is lost. According to the number of exits of the building in the 3D map being m, the other m-1 controllable aircraft terminals nearby are controlled to move to other exits of the building in the 3D map, and at the same time, the m-1 controllable aircraft terminals turn on the far-infrared thermal imagers and use their cameras and far-infrared thermal imagers to move toward other exits of the building. S5: Each controllable aircraft terminal obtains the far-infrared thermal imaging video stream of the newly appeared target at each exit of the building, and compares it with the far-infrared thermal imaging information of the previously obtained tracking target. If the similarity is greater than a preset threshold, the corresponding controllable aircraft terminal acts as a new first tracking aircraft to lock and follow the target; S6: Control other controllable aircraft terminals to fly around the new first tracking aircraft with the new first tracking aircraft as the center.

2. The multi-terminal adaptive data processing method according to claim 1, characterized in that: The data processing method further includes S7: the camera of the new first tracking aircraft obtains the video stream of the target and compares it with the image of the previous tracking target. If the similarity is less than a preset threshold, the image of the tracking target is updated to the latest acquired tracking target image information, and the first tracking aircraft switches to use the updated tracking target image instead of thermal imaging as the tracking object.

3. The multi-terminal adaptive data processing method according to claim 1, characterized in that: The first tracking aircraft tracks the target based on the PID control algorithm according to the camera locking target.

4. The multi-terminal adaptive data processing method according to claim 1, characterized in that: In S3 or S6, when the other controllable aircraft terminals follow the first tracking aircraft around the first tracking aircraft, the flight altitude of the other controllable aircraft terminals is higher than that of the first tracking aircraft, and they respectively lock and track the targets at a long distance.

5. The multi-terminal adaptive data processing method according to claim 4, characterized in that: In S3 or S6, if the first tracking aircraft fails, another controllable aircraft terminal closest to the tracking target is switched to the first tracking aircraft to perform the tracking task originally performed by the first tracking aircraft.

6. A multi-terminal adaptive data processing system, characterized in that: include: The information sending and receiving module of the control terminal is used to send 3D map information within the location range of the tracking target to n controllable aircraft terminals, and receive video stream information collected by the camera and far-infrared thermal imager of each controllable aircraft terminal; The flight control module of each controllable aircraft terminal is used to track the flight according to the 3D map information and the flight instructions sent by the control terminal according to the tracking algorithm; The storage module of the control terminal is used to store 3D map information, as well as tracking target camera images and far-infrared thermal imaging image information in the video stream acquired by the controllable aircraft terminal; The image comparison and recognition module of the control terminal is used to compare the tracking target camera image and far-infrared thermal imaging image information in the video stream obtained by the controllable aircraft terminal with the tracking target camera image and far-infrared thermal imaging image information in the stored video stream to identify the tracking target.

7. A multi-terminal adaptive data processing device, characterized in that: The invention comprises a memory, a processor and a multi-terminal adaptive data processing program stored in the memory and executable on the processor, wherein the multi-terminal adaptive data processing program is configured to implement the steps of the multi-terminal adaptive data processing method according to any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium stores a multi-terminal adaptive data processing program, which, when executed by the processor, implements the steps of the multi-terminal adaptive data processing method according to any one of claims 1 to 5.

Images