Target monitoring system and target monitoring method

By installing a heterogeneous computing platform on unmanned devices and using NPU to process video images, the delay problem caused by wireless signal difference is solved, real-time processing and echo of unmanned devices video images is realized, and it is suitable for a variety of application scenarios.

CN114125380BActive Publication Date: 2024-11-05SENINT(SUZHOU) TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111303116.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-05
Publication Date
2024-11-05
Estimated Expiration
2041-11-05

AI Technical Summary

Technical Problem

In areas where existing unmanned devices have poor wireless signals or limited bandwidth, the cloud processing delay cannot meet the real-time requirements, resulting in untimely video image processing.

Method used

Unmanned equipment is equipped with a heterogeneous computing platform, including a neural network processor NPU chip, to perform real-time processing and recognition of video images, and data transmission is carried out with the controller through wired connections to realize synchronous acquisition, processing and echo of video images.

Benefits of technology

Real-time processing and echo of video images collected by unmanned devices is realized, the size of heterogeneous computing platform is reduced, the needs of various application scenarios are met, and the problem of cloud processing delay is avoided.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114125380B_ABST
    Figure CN114125380B_ABST
Patent Text Reader

Abstract

The present application discloses a target monitoring system and a target monitoring method, which relate to the field of unmanned monitoring technology. The monitoring system includes an unmanned device, a control machine for controlling the unmanned device, and a heterogeneous computing platform connected to the control machine by wire; the unmanned device is used to collect video images and synchronize the video images to the control machine; the control machine is used to forward the video images to the heterogeneous computing platform; the heterogeneous computing platform is equipped with a neural network processor NPU chip, which is used to identify the position of each target in the video image, and feed back the video image after the position is identified to the control machine for echo. The problem that the existing technology may have a delay and thus fail to meet the requirements when processing through the cloud is solved; the effect of processing the video images collected by the unmanned device in real time and realizing real-time echo is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a target monitoring system and a target monitoring method, and belongs to the technical field of unmanned monitoring. Background Art

[0002] In many scenarios, it is often necessary to survey the terrain. For example, to survey a newly discovered forest or to survey wasteland to be reclaimed. In existing solutions, unmanned equipment such as unmanned vehicles or drones are usually used for surveying.

[0003] The current method is that the unmanned equipment is controlled by the PTZ operator, and the unmanned equipment is equipped with a PTZ and a camera. The PTZ operator operates, and the human eye identifies and observes the location of the target area. In the existing solution, the images collected by the unmanned equipment are sent to the cloud for processing and identification. However, in areas with poor wireless signals or limited bandwidth, the network bandwidth of the cloud processing cannot meet the requirements of real-time processing, or there is no cloud connection at all, and the latency cannot meet the requirements. Summary of the invention

[0004] The purpose of the present invention is to provide a target monitoring system and a target monitoring method to solve the problems existing in the prior art.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] According to a first aspect, an embodiment of the present invention provides a target monitoring system, the monitoring system comprising an unmanned device, a control machine for controlling the unmanned device, and a heterogeneous computing platform connected to the control machine by wire;

[0007] The unmanned equipment is used to collect video images and synchronize the video images to the control machine;

[0008] The control machine is used to forward the video image to the heterogeneous computing platform;

[0009] The heterogeneous computing platform is equipped with a neural network processor NPU chip, which is used to identify the position of each target in the video image, and feed back the video image after the position is identified to the control machine for display;

[0010] The heterogeneous computing platform includes: a bus switching and public management module, a central processing unit CPU processing unit, a graphics processing unit GPU processing unit, a neural network processor NPU processing unit, a high-speed data storage unit and a power supply;

[0011] The bus exchange and public management module is used to support the connection, interaction and management of various functional modules;

[0012] The CPU processing unit is used to support the software operation of the main program;

[0013] The GPU processing unit is used to process images and provide image processing acceleration and general computing functions;

[0014] The NPU processing unit is used to provide an acceleration function for the neural network;

[0015] The high-speed data storage unit is used to store large-bandwidth data streams.

[0016] Optionally, the heterogeneous computing platform is also used to highlight each target in the video image.

[0017] Optionally, a binocular camera is provided in the unmanned equipment, and the heterogeneous computing platform is further used to identify the positions of each target in the video image through a binocular ranging algorithm.

[0018] Optionally, the control machine is further used to receive an editing instruction after receiving the echoed video image, and edit the video image according to the editing instruction, wherein the editing instruction is at least one of adding a target, deleting a target, and changing a target position.

[0019] Optionally, the control machine is further used to receive an editing instruction after receiving the echoed video image, and forward the editing instruction to the heterogeneous computing platform, wherein the editing instruction is at least one of adding a target, deleting a target, and changing a target position;

[0020] The heterogeneous computing platform is also used to receive the editing instruction, edit the video image according to the editing instruction, and return the edited video image to the control machine for display.

[0021] Optionally, the heterogeneous computing platform is further used to record the processed video image in a log format after returning the processed video image to the control machine for display.

[0022] In a second aspect, a target monitoring method is provided, the method being used for a heterogeneous computing platform in the monitoring system according to the first aspect, the method comprising:

[0023] Receiving the video image forwarded by the control machine;

[0024] Identifying the location of each target in the video image;

[0025] The video image after the position is identified is fed back to the control machine for display.

[0026] Optionally, a binocular camera is provided in the unmanned device, and the identifying the position of each target in the video image includes:

[0027] The positions of the various targets in the video image are identified by a binocular ranging algorithm.

[0028] Optionally, the method further includes:

[0029] Highlighting each target in the video image;

[0030] Feeding back the video image after the position is identified to the control machine for display includes:

[0031] The video image after identifying the position and highlighting the target is fed back to the control machine for display.

[0032] Optionally, the method further includes:

[0033] Receiving the editing instruction forwarded by the control machine;

[0034] The video image is edited according to the editing instruction, and the edited video image is returned to the control machine for display.

[0035] A target monitoring system is provided, which includes an unmanned device, a control machine for controlling the unmanned device, and a heterogeneous computing platform connected to the control machine by wire; the unmanned device is used to collect video images and synchronize the video images to the control machine; the control machine is used to forward the video images to the heterogeneous computing platform; the heterogeneous computing platform is equipped with a neural network processor NPU chip, which is used to identify the position of each target in the video image, and feed back the video image after the identified position to the control machine for echo. The problem that the existing technology may have a time delay when processing through the cloud and thus cannot meet the requirements is solved; the effect of processing the video images collected by the unmanned device in real time and realizing real-time echo is achieved. At the same time, this application reduces the size of the heterogeneous computing platform by replacing the solution of using multiple CPUs with an NPU, and meets the use of various application scenarios.

[0036] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of the structure of a target monitoring system provided by one embodiment of the present invention;

[0038] Figure 2A schematic diagram of the structure of a heterogeneous computing platform provided for one embodiment of the present invention;

[0039] Figure 3 A schematic diagram of another possible structure of a heterogeneous computing platform provided by an embodiment of the present invention;

[0040] Figure 4 A schematic diagram of the execution of various tasks on a heterogeneous computing platform provided by one embodiment of the present invention;

[0041] Figure 5 A flow chart of a method for operating a target monitoring system provided by an embodiment of the present invention;

[0042] Figure 6 A flowchart of another possible method for operating a target monitoring system provided by an embodiment of the present invention;

[0043] Figure 7 A method flow chart of a target monitoring method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0046] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0047] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0048] Please refer to Figure 1 , which shows a schematic diagram of the structure of a target monitoring system provided by an embodiment of the present application, such as Figure 1 As shown, the monitoring system includes an unmanned device, a control machine for controlling the unmanned device, and a heterogeneous computing platform connected to the control machine by wire;

[0049] The unmanned device may be a drone, an unmanned vehicle or an unmanned ship, and the unmanned device includes a camera and a gimbal, wherein the camera is used to collect surrounding video images, and the gimbal is used to control the route of the unmanned device. In actual implementation, the camera may be a multi-eye camera, for example, a binocular camera, which is not limited in this embodiment.

[0050] This embodiment uses a binocular camera including infrared light and visible light as an example to illustrate that infrared light and visible light can better identify indoor obscured objects and camouflage. In actual implementation, the binocular camera uses a serial connection and can only be turned on or off at the same time, thereby ensuring the timing synchronization of the infrared image and the visible light image, that is, an infrared image and a visible light image can be aligned in timing, that is, at the same time, the aligned visible light image and infrared image of a certain area can be obtained simultaneously.

[0051] By using a binocular camera that can capture both visible and infrared light, the problem in the prior art that it is impossible to accurately obtain infrared and visible light images of the same target at the same time, or that additional computing power and devices are required to achieve timing alignment, is solved.

[0052] In addition, even in the case of partial obstruction, the above binocular camera can still accurately identify the outline of the object, which is a problem that other visible light images cannot solve. Infrared images can obtain the entire outline of the target, and light images can obtain some features of the object. Combining the two, accurate target identification is achieved when most of the target is obscured. Different deep learning networks can be used to train infrared images and visible light images respectively. When judging, the outputs of the two networks are combined to obtain the judgment result, which can obtain a result with higher accuracy than a single visible light or infrared feature. At the same time, this application ensures the computing power of the deep learning network by using a heterogeneous computing platform.

[0053] In addition, the above binocular camera can better identify camouflaged targets. Specifically, if the judgment is based on visible light alone, for example, a human-shaped target, if the image quality is not high and the details are not rich, it is impossible to accurately judge whether it is a real person or a fake person. And if only infrared images are used, it is even more impossible to judge whether it is a human shape. However, through the method of the present application, life forms can be better identified. Using visible light images, the appearance can be identified, such as whether it is a person, and infrared images can be used to identify whether it is a life form. Combining the two, it is easy to judge whether it is a real person or a fake person.

[0054] Typically, the control machine controls the unmanned equipment via radio. In actual implementation, the control machine may be a laptop, a desktop computer or a tablet computer, and, in actual implementation, in order to expand the application scenarios, the control machine described in this application may generally be a portable computer such as a laptop or a tablet computer.

[0055] The unmanned device is used to collect video images and synchronize the video images to the control machine. In actual implementation, the unmanned device synchronizes the collected video images to the control machine according to the image transmission interface protocol. The image transmission interface protocol may be different depending on the specific type of the unmanned device. It only needs to realize video transmission, and its specific protocol type is not limited.

[0056] The control machine is used to forward the video image to the heterogeneous computing platform; the control machine can be connected to the heterogeneous computing platform via a LAN (Local Area Network), such as via a network cable, and after receiving the video image synchronized by the unmanned equipment, the control machine sends the received video image to the heterogeneous computing platform via a wired connection.

[0057] Please refer to Figure 2 The heterogeneous computing platform includes: bus switching and public management modules, CPU (central processing unit) processing unit, GPU (graphics processing unit) processing unit, NPU (Neural-network Processing Unit) processing unit and high-speed data storage unit, as well as power supply. Among them:

[0058] The bus exchange and public management module is responsible for the connection, interaction and management of each functional module, and connects the power module to provide unified external power supply. The heterogeneous computing platform contains multiple heterogeneous computing cores, and each processing unit is physically connected and implements data exchange based on the PCIE (peripheral component interconnect express, high-speed serial computer expansion bus standard) bus. The bus exchange module improves the high-speed data exchange channel, defines the PCIE differential bus signal pins of different units, and uses arc differential routing to reduce electromagnetic scattering and interference to ensure good signal isolation and electromagnetic shielding, improve signal quality and integrity, and ensure stable, reliable and high-speed data communication. The public management module collects information such as temperature, voltage, current, etc. of each module based on the internal I2C bus according to the two-level mode of SMC (public management module) and BMC (each processing unit), and implements resource status supervision of each module.

[0059] The CPU processing unit is used to run the main program software of the heterogeneous computing platform. It has rich external interfaces and 16GB dual-channel DDR3 1066 / 1333 memory for running programs and data buffering. It uses SRIO, PCIE and other interfaces to connect to the heterogeneous computing platform. In addition, it has a display VGA interface, USB interface, network interface, RS232 interface, I2C interface, etc. It can also mount a SATA interface HDD or SSD disk to provide high-performance applications.

[0060] The GPU processing unit is an image processing acceleration core that provides image processing acceleration for various types of payload data and some general computing functions. The GPU has a large-scale processing core and multi-level data cache, built-in massive ALU and floating-point computing units, supports large-scale thread parallel processing, and has greatly improved image processing capabilities compared to the CPU. The main interface parts of the GPU processing unit include: PCIE interface, which is used to communicate with the motherboard and CPU to receive and send data that needs to be processed; clock interface, which generates internal operating clock and drives internal logic operation; I2C and SPI interfaces are mainly used to configure some internal registers; DDR interface, which can connect to 32GB DDR4 memory chips and supports ECC function.

[0061] The NPU processing unit is a deep learning network acceleration core. It is based on a domestic artificial intelligence dedicated main chip and provides powerful deep learning computing capabilities. It supports the rapid deployment of TensorFlow / Caffe deep learning applications. The equivalent theoretical computing power is up to 128TOPS, which can meet the computing and storage requirements of various reasoning scenarios. The main interface parts include: PCIE interface, which is used to communicate with the motherboard and CPU to receive and send data to be processed; clock interface, which generates internal operating clock and drives internal logic operation; I2C and SPI interfaces are mainly used to configure some internal registers; DDR interface, which can connect to 32GB DDR4 memory chip as a cache for running AI algorithms and data, and supports 4-channel 64bit ECC function.

[0062] The NPU processing unit may be an NPU chip, which is used to identify the position of each target in the video image, and feed back the video image after the position is identified to the control machine for display.

[0063] The NPU chip can support the computing power of higher recognition algorithms. In this application, for the same yolov3 recognition algorithm, the recognition speed of the CPU is one frame per second, while the recognition speed of the NPU can reach 10 frames per second. And in actual implementation, the number of targets in each frame of the video image has nothing to do with the recognition speed. Since the image is displayed in real time at a speed of 15 frames per second in actual implementation, under the premise of carrying NPU in this application, when the unmanned device moves at a normal speed, it is possible to recognize the target in the video image.

[0064] The above only uses the example of a heterogeneous computing platform equipped with an NPU. In actual implementation, the heterogeneous computing platform can also be equipped with a CPU to implement control logic.

[0065] In actual implementation, the heterogeneous computing platform can be equipped with different recognition algorithms and recognition models according to the application scenarios, and recognize the video images through the carried recognition algorithms or recognition models. In actual implementation, the NPU can assist in real-time recognition of the relative position of the target in the video image and mark it. In actual implementation, when the camera in the unmanned device is a binocular camera, the heterogeneous computing platform can identify the position of each target in the video image through the binocular ranging algorithm. Of course, in addition to identifying the position, the heterogeneous computing platform can also highlight the appearance of the identified target. Since both the binocular ranging algorithm and the highlighting algorithm are low-computing algorithms, real-time image echo can be achieved in this solution, which solves the problem of delay in the prior art.

[0066] However, since the position obtained by the heterogeneous computing platform based on video image recognition is a relative position, in this embodiment, the unmanned equipment can also calculate the absolute coordinates of each target in the monitored video image based on its own absolute coordinates. The coordinate system of the absolute coordinates can be indoor relative coordinate positioning or GPS (Global Positioning System) positioning, which is not limited in this embodiment.

[0067] The high-speed data storage unit is used to store large bandwidth data streams. It interacts with each processing unit through the Rapid IO high-speed data interface to store data, and implements storage and data interface functions in a concise, efficient and reliable on-chip system. Data capacity is based on demand, and multiple hard disks are configured to achieve reliable data storage.

[0068] The power module provides power and grounding functions, and generates different voltage and current supplies required by each processing unit load on demand. The power module has certain surge resistance, anti-static, low electromagnetic radiation, and good electromagnetic compatibility. It supplies the commonly used power supply voltages of each computing board through DC-DC conversion, and mainly provides independent +5V and +12V power supplies. It supplies +5V and +12V power supplies to each board, and generates the power supply voltages required by different load circuits through the on-board power conversion circuit, such as 1.8V, 3.3V, etc.

[0069] Please refer to Figure 3 The heterogeneous computing platform adopts a bus architecture, and forms a parallel computing environment through multiple data processing modules to complete the real-time processing of load data. It adopts a "multi-level, pipeline, parallel processing, and unified scheduling" solution, follows the PCIE bus interface standard, takes into account computing performance, scalability, and versatility, and supports fault tolerance, reconstruction, and interconnection expansion capabilities. The heterogeneous computing platform adopts a dual-redundant switching interconnection architecture design, and the topology is a dual-star topology. Redundant backup measures are taken for the main nodes or all nodes in the topology, and point-to-point data paths are used between each node module instead of bus paths to help the data processing system improve fault tolerance and avoid module failure that affects the entire system.

[0070] The heterogeneous computing platform is based on a multi-channel, large-capacity, high-speed data processing parallel architecture and adopts a multi-level parallel structure: multiple computing modules are used at the device level to form a distributed storage multiple-instruction multiple-data (MIMD) parallel architecture; intelligent processors and storage are used to form parallel computing nodes at the module level; multi-core processors are used at the intelligent processor chip level, and the on-chip multi-core structure is used to improve the computing power of a single processor.

[0071] In order to maximize the versatility and scalability, the heterogeneous computing platform is based on a distributed storage architecture at the device level. Each computing module in the device has an independent address space, and the modules interact and communicate with each other through the Internet. In order to achieve flexible expansion of the parallel system topology, a high-speed serial bus (SRIO) and a high-speed switching module are used to form an Internet network. By configuring the node forwarding table of the switching module, the topology of the parallel system can be flexibly configured to meet the requirements of different parallel algorithms.

[0072] A parallel computing environment is formed by multiple data processing modules, which are divided into basic functional components and functional load components. Among them, the basic functional components include public management modules, switching control modules and power modules. The switching control module provides a data transmission path for interconnection between modules. Each processing node forms a flexible star topology through the switching module, which is convenient for system expansion; the public management module is responsible for platform system management, realizing fault tolerance, resource management and task scheduling, and completing the dynamic change of the data processing module algorithm. The public management module also realizes the public management service function. The functional load component can expand various functional units, including CPU processing unit, high-speed storage module, GPU processing board and NPU processing board. The CPU processing unit is mainly responsible for the control and scheduling of the load data processing logic; the high-speed storage module is used to store load data, realize the target feature database, and support intelligent computing tasks; the GPU processing board is mainly responsible for the processing and acceleration of the load image data; the NPU processing board is mainly responsible for the acceleration processing of the deep learning network model algorithm.

[0073] Computing tasks on heterogeneous computing platforms include two types of tasks: software tasks and hardware tasks. Software tasks are defined as tasks running on general-purpose processors; hardware tasks are defined as tasks running on heterogeneous computing platforms. Applications are composed of a series of software tasks and hardware tasks, and there can be a calling relationship between tasks. Compared with software tasks, platform hardware tasks have the following characteristics:

[0074] 1) The execution time of hardware tasks is basically fixed. The execution process of hardware tasks can be divided into three stages: input data transmission, task execution, and output data transmission. The task execution stage has a precise cycle, while the time for input and output data transmission depends on the transmission volume and data bandwidth. On heterogeneous computing units, applications suitable for acceleration are streaming data applications, or applications with small data volumes but complex calculations. Streaming data applications usually adopt the method of executing while transmitting, and the data transmission overhead is hidden in the execution process of the task; for applications with small data volumes but complex calculations, the execution time required is much longer than the data transmission time. The execution time of hardware tasks is basically fixed. For streaming applications, it can be deduced from this conclusion that each processing of a certain amount of data is a task. Streaming applications are composed of continuous tasks of the same type, and each of its hardware tasks can be regarded as having a basically fixed execution time.

[0075] 2) To ensure performance and schedulability, hardware tasks usually do not support task preemption. The preemption of software tasks is actually the scheduling of tasks that use processor resources in different time slices according to the scheduling algorithm. When scheduling occurs, its context needs to be saved, including register values, stack and other information. The preemption of hardware tasks requires saving the hardware context, such as the value of the internal register of the heterogeneous computing unit, the execution status and other information. Its overhead is usually much greater than that of the software context, resulting in an increase in the complexity of the heterogeneous computing unit. Moreover, since the format of the hardware context and the software context is not unified, different mechanisms need to be adopted when resuming task execution after preemption. Therefore, the cost of supporting preemption for hardware tasks is high. Usually, hardware tasks do not support task preemption, that is, once a hardware task starts to execute, from the time the task sends data until the calculation result is returned, the heterogeneous computing unit is always exclusively occupied by the hardware task, and other tasks cannot obtain the right to use the heterogeneous computing unit.

[0076] 3) Hardware tasks do not have the ability to communicate directly. Direct communication between hardware tasks requires hardware support, and there must be direct hardware paths between heterogeneous computing units to complete the transmission of signals and data. The platform uses a black box approach to abstract and statically divide resources, and it cannot be changed at runtime. An integer is used to uniquely represent the black box location information, and ultimately corresponds to a unique physical address in the driver. The driver uses the physical address when operating the heterogeneous units associated with the black box, while scheduling and resource management use the black box number for easy management. Heterogeneous computing units that support specific functions can be arranged on each black box. At a certain time, at most one heterogeneous computing unit can be arranged on a running black box, but the arranged heterogeneous computing units can still be adjusted as needed in a specific computing environment, that is, the supported functions of the black box are changed.

[0077] In this application, please refer to Figure 4 ,The application tasks of the heterogeneous computing platform include the following steps:

[0078] Step 1: Heterogeneous task creation. When a batch of heterogeneous tasks appears, the user is first required to provide a task binding algorithm to bind the entire batch of tasks to different computing resources for execution. Software tasks assigned to high-performance processors run on the general host, and tasks assigned to hardware units run on heterogeneous computing units. Tasks on all computing resources are executed in parallel.

[0079] Step 2: Hardware task load: Hardware tasks require the cooperation of general-purpose processors and heterogeneous computing units.

[0080] Step 2.1: Hardware task creation. A task creation request is initiated on the host side and sent to the heterogeneous computing unit side using the interconnect bus. The middleware provides a standard task receiving program for the heterogeneous computing unit side to receive task requests from the general processor side. After receiving the hardware task request, the middleware will create the hardware task and prepare the relevant runtime environment. The hardware task creation process mainly involves the allocation of off-chip memory of the heterogeneous computing unit and the addition of tasks to the hardware task queue.

[0081] Step 2.2: Hardware task scheduling. When the scheduling algorithm selects a hardware task to run, the middleware's runtime preparation for the hardware task is mainly the data copying operation and the initialization of the hardware task parameters. The middleware copies the data to be processed from the general processor end memory to the heterogeneous computing unit off-chip memory according to the artificial intelligence application parameter flag. At the same time, the address of the hardware task parameter in the heterogeneous computing unit memory is assigned to the hardware task parameter.

[0082] Step 2.3: Hardware task execution. The control code on the general processor and the actual execution of the heterogeneous computing unit. When the execution of a given hardware task is completed, the result data is written back to the corresponding memory unit. The task scheduling of the heterogeneous computing unit adopts a first-come, first-served approach. Hardware tasks are added to the ready queue in order. When the current task is completed, the scheduling algorithm selects the next task from the queue to prepare the execution environment.

[0083] Step 3: Software Task Loading Software Tasks Running on General-Purpose Processors All software tasks run on the CPU side.

[0084] Step 3.1: Software task execution. At the end of the task binding algorithm, after sending the task request of the hardware task, the software task starts to execute.

[0085] Step 3.2: Software task check. The executor of the software task completes the software task assigned to it. On the other hand, when the software task is completed, it checks whether the same batch of tasks on other computing resources have been completed to ensure that the work assigned by the binding algorithm has been executed and the data has been synchronized.

[0086] After the heterogeneous computing platform processes the video image, the heterogeneous computing platform can feed back the processed video image to the control machine for display. Correspondingly, the control machine receives the processed video image fed back by the heterogeneous computing platform and displays the video image.

[0087] Optionally, after the control machine performs video echo, the position of the target marked in the video image can be used to prompt the operator of the object in the field of view. At this time, the operator can make a secondary judgment, that is, to judge whether there is a missed target or a misidentified target. That is to say, after the control machine receives the echoed video image, the following steps can also be included: receiving an editing instruction, the editing instruction is at least one of adding a target, deleting a target, and changing the target position.

[0088] For example, when the operator finds that there is an unidentified target in the video image, the operator can click on the unidentified area in the video image; for another example, when an incorrectly identified target is found, the deletion of the target can be triggered.

[0089] After the control machine receives the editing instruction, the control machine can edit it by itself, or send it to the heterogeneous computing platform for editing. That is to say, this embodiment can also include the following two possible implementation methods:

[0090] The first type: the control machine edits the video image according to the editing instruction.

[0091] The second type: the control machine forwards the editing instruction to the heterogeneous computing platform, and the heterogeneous computing platform is also used to receive the editing instruction, edit the video image according to the editing instruction, and return the edited video image to the control machine for display.

[0092] For example, please refer to Figure 5 , which shows a flowchart of the monitoring system when a heterogeneous computing platform edits a manually input target. Figure 5 As shown in the figure, the unmanned equipment collects video images and sends them to the control machine. The control machine receives the images and forwards them to the heterogeneous computing platform through wired transmission such as LAN. The heterogeneous computing platform receives the video, recognizes and processes the video images through model reasoning calculations, and feeds back the processing results to the control machine for display. After the control machine annotates and echoes the recognition results, it receives the operator's manual auxiliary input and forwards the input instructions to the heterogeneous computing platform. The heterogeneous computing platform corrects the video image according to the instructions forwarded by the control machine, and aggregates the processed video image data and feeds it back to the control machine for display.

[0093] As mentioned above, after combining machine judgment and manual correction, the operation of multi-target recognition can be greatly enhanced. In the case of multiple cameras, it can assist in the realization of ranging and positioning operations.

[0094] In addition, the present application does not limit the collection method of the results, which can be logs or screenshots, etc. If the log method is used for recording, a general algorithm or simple rule judgment can be used to remove duplicate targets.

[0095] Please refer to Figure 6 , which shows a complete monitoring flow chart of the target monitoring system provided by the present application. Figure 6 , two groups of heterogeneous computing hardware can perform calculations synchronously. The computing tasks are divided into multiple different grouping tasks. After completion, they are written back according to the set result path and then merged in time sequence. This ensures that the processing results between multiple tasks will not be confused, and heterogeneous computing power realizes synchronous computing and reuses the computing power of the platform. The CPU does not process heterogeneous computing, which ensures the computing power of operations such as input and output user interaction. It will not cause jamming and loss of log files due to intensive computing tasks, thus ensuring user experience. The hardware task adopts timeout-based scheduling. If the frame rate is too high and the computing power is insufficient, the strategy of the hardware task is that if the task stack is full and the new task cannot obtain computing power, then the entire task stack will be cleared and all previous results will be discarded. In this way, even if the echo result may flicker, there will be no obvious lag display, ensuring correctness. Since the two network inferences are calculated on different hardware, the data isolation is physically guaranteed, avoiding the possibility of errors caused by cross scheduling.

[0096] In summary, a target monitoring system is provided, which includes an unmanned device, a control machine for controlling the unmanned device, and a heterogeneous computing platform connected to the control machine by wire; the unmanned device is used to collect video images and synchronize the video images to the control machine; the control machine is used to forward the video images to the heterogeneous computing platform; the heterogeneous computing platform is equipped with a neural network processor NPU chip, which is used to identify the position of each target in the video image, and feed back the video image after the identified position to the control machine for echo. The problem that the existing technology may have a time delay when processing through the cloud and thus cannot meet the requirements is solved; the effect of processing the video images collected by the unmanned device in real time and realizing real-time echo is achieved. At the same time, the present application reduces the size of the heterogeneous computing platform by replacing the solution of using multiple CPUs with an NPU, and meets the use of various application scenarios.

[0097] Please refer to Figure 7 , which shows a method flow chart of a target monitoring method provided by an embodiment of the present application. The target monitoring method described in this embodiment can be used for Figure 1 In the heterogeneous computing platform of the target monitoring system shown in Figure 7 As shown, the target monitoring method includes:

[0098] Step 701, receiving the video image forwarded by the control machine;

[0099] Step 702, identifying the position of each target in the video image;

[0100] Step 703: Feedback the video image after the position is identified to the control machine for display.

[0101] Optionally, a binocular camera is provided in the unmanned device, and the identifying the position of each target in the video image includes:

[0102] The positions of the various targets in the video image are identified by a binocular ranging algorithm.

[0103] Optionally, the method further includes:

[0104] Highlighting each target in the video image;

[0105] Feeding back the video image after the position is identified to the control machine for display includes:

[0106] The video image after identifying the position and highlighting the target is fed back to the control machine for display.

[0107] Optionally, the method further includes:

[0108] Receiving the editing instruction forwarded by the control machine;

[0109] The video image is edited according to the editing instruction, and the edited video image is returned to the control machine for display.

[0110] The specific technical details of each of the above steps are detailed in the above embodiments and will not be repeated here.

[0111] In summary, by receiving the video image forwarded by the control machine; identifying the position of each target in the video image; and feeding back the video image after the position is identified to the control machine for echo. This solves the problem that the existing technology may have a delay and thus fail to meet the requirements when processing through the cloud; it achieves the effect of processing the video images collected by the unmanned equipment in real time and realizing real-time echo. At the same time, this application reduces the size of the heterogeneous computing platform by carrying an NPU instead of using multiple CPUs, meeting the use of various application scenarios.

[0112] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0113] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A target monitoring system, characterized in that: The monitoring system includes an unmanned device, a control machine for controlling the unmanned device, and a heterogeneous computing platform connected to the control machine by wire; The unmanned equipment is used to collect video images and synchronize the video images to the control machine; The control machine is used to forward the video image to the heterogeneous computing platform; The heterogeneous computing platform is equipped with a neural network processor NPU chip, which is used to identify the position of each target in the video image, and feed back the video image after the position is identified to the control machine for display; The heterogeneous computing platform includes: a bus switching and public management module, a central processing unit CPU processing unit, a graphics processing unit GPU processing unit, a neural network processor NPU processing unit, a high-speed data storage unit and a power supply; The bus exchange and public management module is used to support the connection, interaction and management of various functional modules; The CPU processing unit is used to support the software operation of the main program; The GPU processing unit is used to process images and provide image processing acceleration and general computing functions; The NPU processing unit is used to provide an acceleration function for the neural network; The high-speed data storage unit is used to store large bandwidth data streams; The tasks of the application on the heterogeneous computing platform are performed as follows: Heterogeneous task creation: When a batch of heterogeneous tasks appears, the user is first required to provide a task binding algorithm to bind the entire batch of tasks to different computing resources for execution. The software tasks assigned to the high-performance processor run on the general host, and the tasks assigned to the hardware unit run on the heterogeneous computing unit. The tasks on all computing resources are executed in parallel. Hardware task load: Hardware tasks require the cooperation of general-purpose processors and heterogeneous computing units; When executing the hardware task load, the following operations are performed: Hardware task creation: Initiate a task creation request on the host side, and send the request to the heterogeneous computing unit side using the interconnect bus. The middleware provides a standard task receiving program for the heterogeneous computing unit side to receive task requests from the general processor side. After receiving the hardware task request, the middleware will create the hardware task and prepare the relevant runtime environment. The hardware task creation process mainly involves the allocation of off-chip memory of the heterogeneous computing unit and the addition of tasks to the hardware task queue. Hardware task scheduling: When the scheduling algorithm selects a hardware task to run, the middleware's runtime preparation for the hardware task is mainly the data copying operation and the initialization of the hardware task parameters. The middleware copies the data to be processed from the general processor end memory to the heterogeneous computing unit off-chip memory according to the artificial intelligence application parameter flag. At the same time, the address of the hardware task parameter in the heterogeneous computing unit memory is assigned to the hardware task parameter; Hardware task execution, control code on general-purpose processors and actual execution of heterogeneous computing units. When a given hardware task is completed, the result data is written back to the corresponding memory unit. The task scheduling of heterogeneous computing units adopts a first-come, first-served approach. Hardware tasks are added to the ready queue in order. When the current task is completed, the scheduling algorithm selects the next task from the queue and prepares the execution environment. Software Task Load: Software tasks running on general-purpose processors are all run on the CPU side; When executing the software workload, the following actions are performed: Software task execution,At the end of the task binding algorithm, after sending the task request of the hardware task,,the software task starts to execute; Software task check: On the one hand, the executor of the software task completes the software task assigned to it; on the other hand, when the software task is completed, it checks whether the same batch of tasks on other computing resources have been completed to ensure that the work assigned by the binding algorithm has been executed and the data has been synchronized.

2. The monitoring system according to claim 1, characterized in that: The heterogeneous computing platform is also used to highlight each target in the video image.

3. The monitoring system according to claim 1, characterized in that: The unmanned equipment is provided with a binocular camera, and the heterogeneous computing platform is also used to identify the positions of various targets in the video image through a binocular ranging algorithm. The binocular camera supports visible light and infrared light.

4. The monitoring system according to any one of claims 1 to 3, characterized in that: The control machine is further used to receive an editing instruction after receiving the echoed video image, and edit the video image according to the editing instruction, wherein the editing instruction is at least one of adding a target, deleting a target, and changing a target position.

5. The monitoring system according to any one of claims 1 to 3, characterized in that: The control machine is further configured to receive an editing instruction after receiving the echoed video image, and forward the editing instruction to the heterogeneous computing platform, wherein the editing instruction is at least one of adding a target, deleting a target, and changing a target position; The heterogeneous computing platform is also used to receive the editing instruction, edit the video image according to the editing instruction, and return the edited video image to the control machine for display.

6. The monitoring system according to any one of claims 1 to 3, characterized in that: The heterogeneous computing platform is also used to record the processed video image in a log format after returning the processed video image to the control machine for display.

7. A target monitoring method, characterized in that: The method is used for a heterogeneous computing platform in a monitoring system according to any one of claims 1 to 5, and the method comprises: Receiving the video image forwarded by the control machine; Identifying the location of each target in the video image; The video image after the position is identified is fed back to the control machine for display.

8. The method according to claim 7, characterized in that The unmanned device is provided with a binocular camera, and the identifying the position of each target in the video image includes: The positions of the various targets in the video image are identified by a binocular ranging algorithm.

9. The method according to claim 7, characterized in that: The method further comprises: Highlighting each target in the video image; Feeding back the video image after the position is identified to the control machine for display includes: The video image after identifying the position and highlighting the target is fed back to the control machine for display.

10. The method according to claim 7, characterized in that The method further comprises: Receiving the editing instruction forwarded by the control machine; The video image is edited according to the editing instruction, and the edited video image is returned to the control machine for display.