Infrared visible light dual-band image acquisition and processing system based on brain-like calculation

Abstract

The invention discloses an infrared visible light dual-band image acquisition and processing system based on brain-like calculation, and relates to the technical field of image processing. According to the scheme, the system comprises a visible light camera, an infrared camera, an image acquisition unit, a main control unit, an exchange unit and n many-core computing units, the image acquisition unit performs time domain alignment on the infrared image and the visible light image based on timestamps, and sends formed first video stream data to the main control unit; the switching unit receives first video stream data sent by the main control unit and distributes the first video stream data to the many-core computing unit for processing; the many-core calculation unit returns an image processing result to the exchange unit; and the main control unit receives the image processing result sent by the switching unit and generates second video stream data. According to the invention, high-speed synchronous acquisition of the infrared and visible light dual-band image can be realized, and real-time high-speed processing of the infrared and visible light dual-band image can be realized with low energy consumption.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically, to an infrared and visible light dual-band image acquisition and processing system based on brain-like computing. Background Art

[0002] Due to the limited imaging band range of a single image sensor, the information obtained has certain limitations. In some scenes that require complex imaging conditions or full-time operation, single-band sensors (such as visible light sensors) are increasingly unable to meet people's needs for all-round description of scenes and targets in actual use. Since infrared images and visible light images are obviously complementary, and can be imaged under low illumination conditions and have a certain ability to penetrate smoke, more and more applications such as environmental monitoring and target detection and identification introduce infrared and visible light bands to meet the needs of all-day monitoring.

[0003] With the improvement of image resolution and frame rate and the introduction of dual bands, the amount of image data acquired has expanded rapidly, thus placing higher requirements on the speed and effect of infrared / visible light image processing. At the same time, due to the limitations of usage scenarios, such as full-time detection and embedded applications, the energy consumption of the equipment is relatively sensitive and has higher requirements. At present, the commonly used computing platform is usually a general-purpose processor based on the von Neumann architecture, that is, a general-purpose CPU or GPU platform, which has defects such as serious resource consumption, slow operation speed, and high energy consumption. As a result, it can only rely on server clusters to provide cloud computing services, and it is difficult to apply to mobile terminals or embedded devices that have high requirements for area, speed, and energy consumption. Summary of the invention

[0004] In view of this, the present invention provides an infrared-visible light dual-band image acquisition and processing system, which can realize high-speed synchronous acquisition of infrared-visible light dual-band images, and perform real-time high-speed processing of infrared-visible light dual-band images with low energy consumption.

[0005] In order to achieve the above object, the present invention adopts the following technical scheme:

[0006] An infrared and visible light dual-band image acquisition and processing system based on brain-like computing includes: a visible light camera, an infrared camera, an image acquisition unit, a main control unit, a switching unit, and n many-core computing units; wherein,

[0007] A visible light camera is connected to the image acquisition unit and is used to acquire a visible light image and send the visible light image to the image acquisition unit;

[0008] An infrared camera, connected to the image acquisition unit, is used to acquire infrared images and send the infrared images to the image acquisition unit;

[0009] An image acquisition unit is connected to the visible light camera, the infrared camera and the main control unit respectively, and is used to receive the visible light image sent by the visible light camera and the infrared image sent by the infrared camera, respectively obtain the timestamps of the infrared image and the visible light image, and determine the infrared image and the visible light image whose timestamp difference is less than a preset time length as a time domain aligned image pair, and send the first video stream data containing multiple time domain aligned image pairs to the main control unit; wherein the timestamp difference is the absolute value of the difference between the timestamp of the infrared image and the timestamp of the visible light image;

[0010] The main control unit is connected to the image acquisition unit and the switching unit respectively, and is used to receive the first video stream data sent by the image acquisition unit and send the first video stream data to the switching unit;

[0011] a switching unit, connected to the main control unit and the n many-core computing units respectively, for receiving the first video stream data sent by the main control unit, and sequentially generating a plurality of batches of image processing tasks based on the first video stream data, and distributing each batch of image processing tasks to the n many-core computing units in parallel; wherein each batch of image processing tasks includes n subtasks, the subtasks correspond to the many-core computing units one by one, each subtask includes k image pairs, k≥1, n≥1, and k and n are both integers;

[0012] The many-core computing unit includes at least one brain-like computing chip, which is used to receive and use the brain-like computing chip to process k image pairs in the subtask distributed by the switching unit, and return the image processing results to the switching unit; wherein the brain-like computing chip is deployed with an algorithm network model, and the deployment steps of the algorithm network model are: network parameter integerization, network topology structure mapping and chip physical layout of the algorithm network model, and adjustment of the algorithm network model; then the brain-like computing chip is initialized, chip configuration and initialization files are generated, downloaded to the chip system, and the brain-like computing chip is started to run;

[0013] The exchange unit is further used to send the image processing results of the current batch of subtasks returned by the many-core computing units to the main control unit, and after all the many-core computing units have returned the image processing results of the current batch of subtasks, send the next batch of image processing tasks to the many-core computing units;

[0014] The main control unit is also used to receive the image processing result of the multi-core computing unit on the subtask sent by the switching unit, and generate the second video stream data based on the image processing result of the subtask.

[0015] In a possible implementation, the value interval of k is [1,3].

[0016] In one possible implementation, the many-core computing unit is used to select k brain-like computing chips to process the k image pairs in the subtask if the number k of image pairs in the subtask is less than the number of brain-like computing chips; if the number k of image pairs in the subtask is not less than the number of brain-like computing chips, then all brain-like computing chips are selected to process the k image pairs in the subtask.

[0017] In a possible implementation, the method further includes:

[0018] The camera control module is connected to the main control unit, the infrared camera and the visible light camera respectively, and is used to receive and respond to the camera control instructions sent by the main control unit to control the infrared camera and the visible light camera.

[0019] In a possible implementation, the method further includes:

[0020] The display module is connected to the main control unit and is used for receiving and displaying the second video stream data sent by the main control unit.

[0021] In a possible implementation, the method further includes:

[0022] The storage module is connected to the main control unit and is used for receiving and storing the first video stream data and the second video stream data sent by the main control unit.

[0023] In a possible implementation, the method further includes:

[0024] It also includes an external debugging interface, which is connected to the main control unit and is used to receive external instructions sent by an external debugging device and send the external instructions to the main control unit; wherein the external instructions include debugging instructions, maintenance instructions, data exchange instructions and algorithm deployment instructions.

[0025] Compared with the prior art, the present invention realizes the simultaneous acquisition of infrared and visible light dual-band images, and can realize the alignment of the two in the time domain, and uses a standard network interface for output to support gigabit network for image data transmission and control command exchange. The application of dual bands can make up for the poor performance of visible light images at night and under complex imaging conditions, as well as the weaknesses of infrared images such as low resolution and weak texture. It combines the advantages of dual bands and makes up for the shortcomings to achieve all-weather and all-time monitoring, and helps to improve the algorithm accuracy of advanced visual tasks such as tracking and segmentation. Compared with traditional CPU and GPU computing platforms, the use of brain-like computing platforms as the terminal for real-time image processing greatly improves the energy consumption ratio, making it more suitable for the application of artificial intelligence algorithms in terminal devices such as embedded and mobile terminals. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0027] Figure 1 It is a structural schematic diagram of an infrared and visible light dual-band image acquisition and processing system based on brain-like computing in an embodiment of the present invention;

[0028] Figure 2 is a schematic diagram of a workflow of an image acquisition unit according to an embodiment of the present invention;

[0029] Figure 3 This is a minimum functional core architecture diagram of a brain-like computing chip according to an embodiment of the present invention;

[0030] Figure 4 is a schematic diagram of an artificial neural network algorithm configuration process according to an embodiment of the present invention;

[0031] Figure 5 It is a schematic diagram of an artificial neural network streaming topology mapping process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0033] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0034] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0035] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0036] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0037] The embodiment of the present invention provides an infrared and visible light dual-band image acquisition and processing system based on brain-like computing. Figure 1 , which is a schematic diagram of the structure of the infrared and visible light dual-band image acquisition and processing system based on brain-like computing provided by an embodiment of the present invention. Figure 1As shown, the infrared and visible dual-band image acquisition and processing system based on brain-like computing provided by the present invention mainly includes a visible light camera 101, an infrared camera 102, an image acquisition unit 103, a main control unit 104, a switching unit and n many-core computing units 106, wherein n≥1 and n is an integer.

[0038] In this embodiment, the visible light camera 101 is connected to the image acquisition unit 103 for acquiring visible light images and sending the visible light images to the image acquisition unit 103 ; the infrared camera 102 is also connected to the image acquisition unit 103 for acquiring infrared images and sending the infrared images to the image acquisition unit 103 .

[0039] The infrared camera 102 and the visible light camera 101 are embedded in the gimbal with their optical axes parallel. The housing of the gimbal can be sealed and waterproof. The gimbal can rotate 360° horizontally and 90° vertically. The camera data transmission line is connected to the camera acquisition module through the aviation socket, which is not easy to cause poor contact of the wire due to vibration and shaking. It also has rainproof function, which is conducive to achieving all-weather detection tasks. Four standard screw holes are designed at the bottom of the gimbal for easy installation and deployment.

[0040] The system further includes a camera control module 107 connected to the main control unit 104, the infrared camera 102 and the visible light camera 101 respectively, which is mainly composed of a joystick and a control keyboard. The joystick controls the rotation of the camera pan, and the control keyboard controls the focal length and working mode of the infrared camera 102 and the visible light camera 101. The camera control module 107 can directly control the infrared camera 102 and the visible light camera 101 through the RS232 protocol (Recommended Standard 232, a serial communication protocol), and can also be connected to the main control unit 104, receive and respond to the camera control instructions sent by the main control unit 104 through the network protocol, and control the infrared camera 102 and the visible light camera 101.

[0041] In this embodiment, the image acquisition unit 103 is connected to the visible light camera 101, the infrared camera 102 and the main control unit 104 respectively, receives the visible light image sent by the visible light camera 101, and receives the infrared image sent by the infrared camera 102, performs temporal frame alignment on the infrared image and the visible light image, and forms the first video stream data. Specifically, while receiving the infrared image and the visible light image, the image acquisition unit 103 also obtains the respective timestamps of the infrared image and the visible light image. If the absolute value of the difference between the two timestamps is less than a preset time length (for example, 2ms), it can be determined that the infrared image and the visible light image are a time domain aligned image pair, and the first video stream data containing multiple time domain aligned image pairs is sent to the main control unit 104.

[0042] The image acquisition unit 103 exchanges data with the main control unit 104 through network protocols, such as IPv4 / IPv6 (Internet Protocol Version 4 / Internet Protocol Version 6), FTP (File Transfer Protocol), RTSP (Real Time Streaming Protocol, a network control protocol), TCP (Transmission Control Protocol, a connection-oriented, reliable, byte stream-based transport layer communication protocol) and other protocols.

[0043] The image acquisition unit 103 is connected to the visible light camera 101 and the infrared camera 102 through a network port. Its core component is FPGA (Field-Programmable Gate Array), which can realize functions such as image acquisition and image data acceleration. Figure 2 As shown, the FPGA workflow is as follows:

[0044] (1) The FPGA sends a UDP (User Datagram Protocol) broadcast packet to its subnet to search for the connected infrared camera 102 and visible light camera 101;

[0045] (2) FPGA waits for the infrared camera 102 and the visible light camera 101 to return a response packet, and extracts the IP address and MAC address of the infrared camera 102 and the visible light camera 101 from the response packet as the destination address in the IP header and MAC header of the sent data packet;

[0046] (3) FPGA sends a write register instruction packet to sequentially configure the functional registers of the infrared camera 102 and the visible light camera 101 and set their working parameters;

[0047] (4) FPGA sends a write acquisition enable register instruction packet to trigger the infrared camera 102 and the visible light camera 101 to acquire and transmit images;

[0048] (5) The FPGA receives camera data packets from the infrared camera 102 and the visible light camera 101, and extracts valid image data and acquisition timestamps therefrom.

[0049] (6) The FPGA performs time alignment on the decoded infrared image and visible light image according to the timestamp, and combines and outputs them to the lower main control unit 104.

[0050] Optionally, the FPGA of the image acquisition unit 103 may adopt Xilinx Artix-7 (XC7A100T), which has sufficient logic resources to process dual-channel video streams and supports a high-speed external memory interface DDR3 as a frame buffer.

[0051] In this embodiment, the switching unit is connected to the main control unit 104 and the n many-core computing units 106 respectively, receives the first video stream data sent by the main control unit 104, and generates multiple batches of image processing tasks in sequence based on the first video stream data, and distributes each batch of image processing tasks to the n many-core computing units 106 in parallel; wherein each batch of image processing tasks includes n subtasks, the subtasks correspond one-to-one to the many-core computing units 106, and each subtask includes k image pairs, k≥1, and k is an integer.

[0052] For the value of k, the principle that the time required for image acquisition of each batch is approximately equal to the processing time of a single image pair can be adhered to, so that the overall operation efficiency of the system is maximized. Specifically, the algorithm can be debugged by setting different k values ​​to obtain the optimal value of k.

[0053] Optionally, the value of k is in the range [1,3].

[0054] It should be noted that each image pair in the first video stream data carries its own timestamp information, wherein the timestamp of the image pair can be obtained by taking the average of the timestamps of the infrared image and the visible light image that constitute the image pair. Therefore, when the switching unit allocates the image pairs to each many-core computing unit 106, it can allocate them in the order of timestamps. For example, k consecutive image pairs are first allocated to the first many-core computing unit 106, and then the subsequent k consecutive image pairs are allocated to the second many-core computing unit 106, and so on. Of course, n consecutive image pairs can also be allocated to n many-core computing units 106 respectively, and then the subsequent n consecutive image pairs are allocated to n many-core computing units 106 respectively, and so on.

[0055] Preferably, the switching unit is a PCIE (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard) switching unit 105, which distributes image processing tasks to multiple many-core computing units 106 in parallel in the form of PCIE protocol and receives data processed by each many-core computing unit 106. It can convert the PCIe channels provided by the CPU and the motherboard chipset into more PCIe expansion interfaces. By connecting multiple PCIe ports to a high-speed switch chip, more PCIe channels can be realized, thereby supporting the connection of more PCIe devices.

[0056] Optionally, the core components of the PCIE switching unit 105 are a high-speed switch core PEX89144 and 8 PCIE interfaces. When multiple many-core computing units 106 are connected to the PCIE switching unit 105, the PCIE switching unit 105 will forward data from one port to another as needed to achieve data transmission between multiple devices. In this way, the PCIE switching unit 105 can connect multiple PCIe channels to a high-speed switch chip, thereby achieving more PCIe channels. The PCIe switch routes and forwards data by organizing the exchange between PCIe channels within itself, and transmits data between the many-core computing units 106 connected to the PCIE switching unit 105. In this process, the PCIE switching unit 105 plays a role similar to a router, directly connecting the data transmission from the main control unit 104 to each many-core computing unit 106, making full use of the limited PCIe channels of the computing units and maximizing the utilization of PCIe channel resources.

[0057] Each multi-core computing unit 106 can be connected to the PCIe 4X interface of the PCIE switching unit 105. The PCIe 4X interface has 4 channels, each of which is full-duplex and can independently send and receive data. These channels can be regarded as 4 independent communication lines, which work together to provide a higher data transmission rate.

[0058] In this embodiment, the many-core computing unit 106 includes at least one brain-like computing chip, which is used to receive and use the brain-like computing chip to process k image pairs in the subtasks distributed by the exchange unit, and perform feature extraction, feature point matching, image registration, image fusion, target detection and other processing on the image pairs in sequence to generate a final image, and return the final image to the exchange unit as an image processing result.

[0059] When the number k of image pairs in a subtask is less than the number of brain-like computing chips included in the many-core computing unit 106, k brain-like computing chips are selected therefrom to process the k image pairs in the subtask respectively, and there are idle brain-like computing chips at this time.

[0060] When the number k of image pairs in a subtask is not less than the number of brain-like computing chips contained in the many-core computing unit 106, all brain-like computing chips are selected to process the k image pairs in the subtask. At this time, each brain-like computing chip must process an image pair, and there are brain-like computing chips that need to process multiple image pairs.

[0061] After receiving the image processing results of the subtasks returned by the many-core computing unit 106 , the switching unit sends them to the main control unit 104 , and the main control unit 104 sorts and splices these image processing results according to the timestamps to obtain the second video stream data.

[0062] After all the many-core computing units 106 return the image processing results for the subtasks of the current batch, the exchange unit sends the next batch of image processing tasks to the many-core computing units 106 .

[0063] It should be noted that Figure 1 Only four many-core computing units 106 and an example in which the many-core computing unit 106 is composed of two brain-like computing chips are shown. The number of many-core computing units 106 and brain-like computing units is set according to actual conditions.

[0064] By designing the processing logic of the image acquisition unit 103, the problem of high-speed synchronous acquisition of infrared and visible light dual-band cameras is solved. Since the brain-like computing chip is a non-von Neumann architecture design that integrates storage and computing, the power consumption and bandwidth required for algorithm implementation are significantly reduced, and the computing efficiency is greatly improved. Therefore, the network parameters can be sparse and quantized without losing network performance; at the same time, the storage and calculation of weight parameters are collaboratively optimized to reduce power consumption when accessing storage, and the characteristics of neurodynamics are used to improve computing efficiency through hardware design. At the same time, multiple brain-like computing chips are designed to work together through the PCIE switching unit 105 to solve the problem of high energy consumption of image processing after image acquisition and reduce computing power consumption and bandwidth.

[0065] In this embodiment, the system may further include a display module (not shown) connected to the main control unit 104 , and configured to receive and display the second video stream data sent by the main control unit 104 .

[0066] Optionally, the display module can be connected to the main control unit 104 via an HDMI cable to display the second video stream data processed by the many-core computing unit 106. Of course, other communication methods can also be used, depending on the communication methods supported by the display module and the main control unit 104. If the display module and the main control unit 104 support different communication methods, they can also be connected via a data adapter cable.

[0067] In this embodiment, the system may further include a storage module 108 connected to the main control unit 104 , which is used to receive and store the first video stream data and the second video stream data sent by the main control unit 104 .

[0068] Optionally, the storage module 108 may be connected to the main control unit 104 via a SATA (Serial Advanced Technology Attachment) interface to store the video stream data before and after being processed by the many-core computing unit 106 .

[0069] In this embodiment, the main control unit 104 is connected to the image acquisition unit 103 and the switching unit respectively, and sends the first video stream data obtained from the image acquisition unit 103 to the switching unit. Its core component is the low-power x86 architecture processor Rockchip RK3568, and it also has two DDR4 memory controllers, with a total memory capacity of 32GB and a frequency of 3200MT / s.

[0070] The main control unit 104 is responsible for connecting and transmitting data with the image acquisition unit 103, the storage module 108, the display module, the camera control module 107, and the switching unit, aggregating, managing, and maintaining data of each module, performing timing control and process management on image processing tasks, monitoring the operating status information of each module during operation, and generating log records based on the operating status information; shutting down the faulty module, and controlling the module to reset and retry after the fault is eliminated; and performing overvoltage protection, undervoltage protection, and overcurrent protection on each module.

[0071] In some optional embodiments, the system also includes an external debugging interface (not shown) connected to the main control unit 104, which is used to receive external instructions sent by an external debugging device and send the external instructions to the main control unit 104; wherein the external instructions include debugging instructions, maintenance instructions, data exchange instructions and algorithm deployment instructions.

[0072] Specifically, the external debugging interface can be set to a USB interface, such as a USB3.0 interface. The selection of the interface type is related to the communication interface of the external debugging device. The number of external debugging interfaces can be set to one or more, which is related to actual external debugging requirements.

[0073] Furthermore, the system may also be provided with one or more indication buttons (not shown) connected to the main control unit 104, for indicating the device status of the main control unit 104 in executing each external command. Specifically, the indication buttons correspond to the external commands one by one, and the light colors of each indication button may be different or the same.

[0074] The hardware design of the overall system adopts a modular design concept, effectively connecting various components through standardized interfaces and reliable connectors, and controlling the overall volume through an integrated chassis structure design. Each functional module in the equipment can be replaced, upgraded and maintained independently, achieving a small structure, easy expansion and upgrade of functions, and easy maintenance of the equipment platform.

[0075] Artificial intelligence algorithms and brain-like intelligence models for visual tasks such as target recognition and tracking, semantic segmentation, etc. can be deployed in brain-like computing chips. Multiple many-core computing units 106 coordinate computing in parallel to complete specific task requirements.

[0076] When deploying artificial neural network algorithms, a mapping method is used based on the minimum functional core of brain-like computing. Figure 3 As shown in the figure, its main structure includes axons, synapses, dendrites, cell bodies and routing networks. The axon unit is equivalent to the input cache, which is mainly used to cache the front-end input information of neurons and provide the required input data for the neuron calculation in the next time step; the synapse array mainly completes the storage of connection topology and weight parameters, and establishes access data interfaces with other units that need to access connections and weights; the dendrite unit mainly completes the multiplication and accumulation of inputs and weights; the cell body unit mainly performs nonlinear processing on the output; the routing unit is mainly used for information transmission between neurons, that is, for configuring the network topology.

[0077] Brain-like computing chips use fixed-point numbers to calculate network parameters to reduce the overly complex floating-point calculation process. At the same time, since it is impossible to assign too large an input weight to a single neuron and the hardware often has communication bit width constraints, the artificial neural network often needs to perform a series of operations such as network integerization when it is mapped to the brain-like chip for calculation in order to achieve maximum computing efficiency. The mapping process of the artificial neural network (ANN) in the brain-like computing chip is as follows: Figure 4 shown.

[0078] The mapping of algorithm networks mainly includes network shaping and network topology mapping, which are mainly used to configure weight parameters such as cell bodies, synapses, routing tables and timing in brain-like computing chips.

[0079] Network integerization is mainly to solve the limitations of storage and communication bit width and computing accuracy in brain-like computing chips. Usually, the floating-point network model is converted into 8b or 10b signed integer data through table lookup and chain scaling rules to meet the computing limitations of brain-like chips. However, due to the centralized multi-core parallel processing architecture, the computing resources of a single core are limited and the information is local. Therefore, the network topology and the calculation process require global control, and the number of nodes in the usual network model is far more than the range supported by a single minimum computing unit. Therefore, in the process of network mapping, the network needs to be split and mapped to a chip or chip group according to the hardware architecture. The specific convolution mapping process is as follows: Figure 4 and Figure 5As shown in the figure, each minimum computing unit FCore is divided into three functions: caching, sorting, and computing. The output of the cache FCore is fed back to the sorting FCore across rows, and the sorting FCore is responsible for collecting and sorting the input FM of the historical rows. After sorting, it is sent to the computing FCore. The computing FCores share the output of the sorting FCore in a direct multicast manner and are responsible for completing convolution and pooling operations. All FCores are configured in multi-phase mode, and the number of phases is connected to each other. Each FCore dendrite accumulation and cell body calculation can be started in the phase that needs to be calculated, and closed in other phases. The cooperation allows the input Feature Map to be input by row and output in a pipeline.

[0080] The original network model can be configured and used in the brain-like chip according to the following steps:

[0081] Step 1: Establish the upper original network model.

[0082] Step 2: Parameter configuration and chip layout - mainly includes network parameter shaping, network topology mapping and chip physical layout. The first two refer to cell body parameter configuration, synapse parameter configuration, routing parameter configuration and working mode, timing and related parameter configuration. If it does not meet the application requirements after comprehensive consideration of factors such as resource consumption and network performance, return to step 1 to adjust the original network model; otherwise, continue.

[0083] Step 3: Initialize the chip status, generate chip configuration and initialization files, download them to the chip system, and start the chip operation.

[0084] The system has the characteristics of low power consumption, high performance, and ease of use, and can run artificial neural network algorithms efficiently and in real time. The overall power consumption is 60w, and under half-precision floating point numbers, it can simulate 250,000 neuron trees, and the energy consumption ratio (synaptic operations / W) is 1TSops / W. Under half-precision floating point numbers, the computing power is 16TFlops, and the computing power of 8-bit bytes is 32Tops.

[0085] The system proposed in this embodiment solves the problems of low computing power density, high power consumption, fixed computing power functions, difficult performance expansion, difficult upgrade and maintenance of traditional computing power platforms such as CPU+GPU, and provides a passive heat dissipation waterproof brain-like processing system with flexible architecture. The system has the characteristics of high computing power density, high performance, low power consumption, compact structure, tailorable and expandable functional performance and upgradeable system hardware. The image acquisition, image data storage, image data calculation and artificial intelligence reasoning acceleration functions of infrared camera 102 and visible light camera 101 can be realized on a single platform, which can meet the use and deployment requirements of high-performance image acquisition and processing in harsh environments.

[0086] Although some specific embodiments of the present invention have been described in detail by way of example, it will be appreciated by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It will be appreciated by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. An infrared and visible light dual-band image acquisition and processing system based on brain-like computing, characterized in that: include: Visible light camera, infrared camera, image acquisition unit, main control unit, switching unit and n many-core computing units; among which, The visible light camera is connected to the image acquisition unit and is used to acquire a visible light image and send the visible light image to the image acquisition unit; The infrared camera is connected to the image acquisition unit and is used to acquire infrared images and send the infrared images to the image acquisition unit; The image acquisition unit is connected to the visible light camera, the infrared camera and the main control unit respectively, and is used to receive the visible light image sent by the visible light camera and the infrared image sent by the infrared camera, respectively obtain the timestamps of the infrared image and the visible light image, and determine the infrared image and the visible light image whose timestamp difference is less than a preset time length as a time domain aligned image pair, and send the first video stream data containing a plurality of the time domain aligned image pairs to the main control unit; wherein the timestamp difference is the absolute value of the difference between the timestamp of the infrared image and the timestamp of the visible light image; The main control unit is connected to the image acquisition unit and the switching unit respectively, and is used to receive the first video stream data sent by the image acquisition unit, and send the first video stream data to the switching unit; The switching unit is connected to the main control unit and the n many-core computing units, respectively, and is used to receive the first video stream data sent by the main control unit, and sequentially generate multiple batches of image processing tasks based on the first video stream data, and distribute each batch of image processing tasks to the n many-core computing units in parallel; wherein each batch of image processing tasks includes n subtasks, and the subtasks correspond to the many-core computing units one by one, and each of the subtasks includes k image pairs, k≥1, n≥1, and k and n are both integers; The many-core computing unit includes at least one brain-like computing chip, which is used to receive and use the brain-like computing chip to process k image pairs in the subtask distributed by the switching unit, and return the image processing results to the switching unit; wherein the brain-like computing chip is deployed with an algorithm network model, and the deployment steps of the algorithm network model are: network parameter integerization, network topology structure mapping and chip physical layout of the algorithm network model, and adjustment of the algorithm network model; then the brain-like computing chip is initialized, a chip configuration and initialization file is generated, downloaded to the chip system, and the brain-like computing chip is started to run; The exchange unit is further configured to send the image processing results of the current batch of subtasks returned by the many-core computing units to the main control unit, and after all the many-core computing units return the image processing results of the current batch of subtasks, send the next batch of image processing tasks to the many-core computing units; The main control unit is further used to receive the image processing result of the subtask by the many-core computing unit sent by the switching unit, and generate second video stream data based on the image processing result of the subtask.

2. The infrared and visible dual-band image acquisition and processing system based on brain-like computing according to claim 1 is characterized in that: The value interval of k is [1,3].

3. The infrared and visible dual-band image acquisition and processing system based on brain-like computing according to claim 1 is characterized in that: The many-core computing unit is configured to select k brain-like computing chips to process the k image pairs in the subtask if the number k of image pairs in the subtask is less than the number of brain-like computing chips; If the number k of image pairs in the subtask is not less than the number of brain-inspired computing chips, all brain-inspired computing chips are selected to process the k image pairs in the subtask.

4. The infrared and visible dual-band image acquisition and processing system based on brain-like computing according to any one of claims 1 to 3, characterized in that: Also includes: The camera control module is connected to the main control unit, the infrared camera and the visible light camera respectively, and is used to receive and respond to the camera control instructions sent by the main control unit to control the infrared camera and the visible light camera.

5. The infrared and visible dual-band image acquisition and processing system based on brain-like computing according to any one of claims 1 to 3, characterized in that: Also includes: The display module is connected to the main control unit and is used to receive and display the second video stream data sent by the main control unit.

6. The infrared and visible dual-band image acquisition and processing system based on brain-like computing according to any one of claims 1 to 3, characterized in that: Also includes: A storage module is connected to the main control unit and is used to receive and store the first video stream data and the second video stream data sent by the main control unit.

7. The infrared and visible dual-band image acquisition and processing system based on brain-like computing according to any one of claims 1 to 3, characterized in that: Also includes: It also includes an external debugging interface, which is connected to the main control unit and is used to receive external instructions sent by an external debugging device and send the external instructions to the main control unit; wherein the external instructions include debugging instructions, maintenance instructions, data exchange instructions and algorithm deployment instructions.

Images