Real-time target detection

Abstract

A system for real-time detection of targets comprises a video recording device, which is a thermal imaging module based on a digital signal processor, including a computing unit (4) having a system-level chip, a RAM chip and a FLASH chip, a power supply system (5) composed of a pulse converter and a power management integrated circuit for configuring the voltage required by all circuit board components, and a data recording component (9, 10, 11) for recording data on an SD card (12), configured in the form of an expansion board. MMC and UART signals come from the power supply circuit board, and the UART is converted into an RS-232 standard signal. An analog converter unit (3) in the form of an input power stage composed of a low voltage drop linear regulator, and an Ethernet interface unit (6) are also provided. The analog converter unit is used to amplify and digitize the video signal from the thermal imaging array (8). The present invention aims to improve the accuracy of target detection in a thermal imaging module with a non-cooled array and to increase the speed of the processor.

Description

[0001] One group of inventions, “Detecting targets in real time”, allows for intelligent image processing (a target recognition algorithm) to be implemented on a processor inside a thermal imaging module and combines video recording and video content analysis in one device. Technical Field

[0002] This group of inventions relates to electronic instruments and is intended to be widely used in the fields of safety systems, ADAS (advanced driver assistance systems), mobile robots (including UAVs (unmanned aerial vehicles)), etc. Background Art

[0003] A known utility model is a thermal imaging device (patent document RU184553U1). The utility model relates to the field of optoelectronic instruments. The thermal imaging device comprises a multi-element photoelectric detector, a lens for forming an infrared image on the plane of the sensitive element of the multi-element photoelectric detector, and a combination module with a unified interface, including a signal preprocessing module, a control module, an electronic processing module, a power module and an interface module. The thermal imaging device comprises a bearing bracket with a detachable bracket and a fastening screw, a sleeve with an outer diameter corresponding to the inner diameter of the bracket and an inner thread corresponding to the lens mounting thread, a shutter, and a threaded frame with an outer thread corresponding to the inner thread of the sleeve. The bearing bracket is provided with a sleeve with a lens mounted thereon and a threaded frame with a shutter mounted thereon. The multi-element photoelectric detector and the signal preprocessing module are located on a first board, the control module and the electronic processing module are located on a second board, and the power supply and interface module are located on a third board. The threaded frame and the end face of the first board are provided with evenly distributed holes. The holes on the frame are threaded, wherein screws for fixing the first board are mounted therein. The first, second and third boards have evenly distributed openings at their corners and have studs for fastening the first, second and third boards mounted in sequence through cylindrical bushings. The first board faces the receiving pads of the multi-element photodetector facing the lens, and the boards are electrically connected through sandwich connectors located thereon. The technical effect is to improve the technical characteristics of the TID: improve image clarity, reduce the weight and volume of the TID, reduce the minimum allowable temperature difference, improve image quality, improve performance, and ensure the ability to work in the thermal range of 8...14μm.

[0004] Compared with the technical solution proposed in the present invention, its disadvantage is that the signal processing process is not perfect enough.

[0005] There is a known invention, namely a unified thermal imaging device (patent document RU2420770C1). The device comprises a multi-element photodetector, an optical system for forming an infrared image in the plane of the sensitive element of the multi-element photodetector, a cooling system and an information processing unit integrated on a single support frame, and a video monitoring unit for visualizing the thermal image. The integrated elements are designed as modules with a unified interface. The integrated module comprises a signal preprocessing module, a cooling system control module, a control interface module, a control device and a power supply module. The information processing unit is designed as an electronic processing module. The electronic processing module provides communication between the signal preprocessing module, the cooling system control module, the control interface module, the power supply module and the video monitoring unit. The control unit is connected to the control interface module and the video monitoring unit.

[0006] One disadvantage of the invention is the excessive number of modules in the system, which slows down signal processing, compared to the solution proposed by the present invention, which has fewer processing units but operates more efficiently.

[0007] There is a known invention, namely, a reticle generation and control device (patent document RU2700034C2). The invention relates to optoelectronic technology and can be used in various aiming and observation devices equipped with optical and thermal imaging aiming and observation channels. The device comprises a ballistic mechanism designed for inputting aiming angles in an optical channel, an analog potentiometer, a low-pass filter, a microcontroller and a grid mover with its own control unit, a thermal imaging aiming and observation channel. The thermal imaging module and the microdisplay are used to generate aiming marks and scales of the thermal imaging channel on the microdisplay and provide control of the marks and scales. For digital video transmission, an LVDS / TTL video converter, a programmable logic device (PLD), a microcontroller and a random access memory (RAM) are used; for analog video transmission, a burst separator, RAM, a microcontroller, a PLD and an aiming mark and reticle adding circuit are used. The invention eliminates manual control of aiming angle input, improves the accuracy of aiming mark and scale alignment, and improves the generation of aiming marks and scales and the automatic control of marks and scales in the optoelectronic path.

[0008] The disadvantage is that PLD circuits are used, while the newly developed solution of the present invention uses a processor, so the system performance is improved.

[0009] The closest analog to the claimed device is described in patent US2016239936A1 (18 / 08 / 2016): a multi-channel image signal processing unit. The image signal processing unit includes: a channel converter for separating an input signal stream including an image signal generated by a plurality of pixels into a processing unit and generating a plurality of processing unit signals; an image signal processing core including a plurality of channel image processing units, each channel image processing unit performing an image signal processing operation and generating a plurality of single output signals by receiving and processing a plurality of single output signals passing through one or more image processing channels in parallel; a channel combiner for combining a plurality of single output signals and generating an output signal stream; a configuration controller for controlling at least one of the plurality of processing unit signals according to an operation mode and selecting a frequency of a processing unit clock signal; and a processing controller for selecting the frequency of the plurality of processing unit signals.

[0010] The disadvantage is that signal processing and conversion require a large number of channels, while the newly developed solution implements the computationally complex recognition algorithm directly on the processor of the thermal imaging module's computing unit. This greatly simplifies the integration of the newly developed thermal imaging module into a system using video content analysis systems and improves the reconfigurability and scalability of these systems.

[0011] A cascade structure of the recognition algorithm consisting of a unique combination of components was developed to significantly increase the speed of the recognition algorithm while maintaining high quality performance.

[0012] A new method for generating HOG descriptors of sliding window frames is developed, and then the descriptors are classified using a linear SVM classifier.

[0013] A new method for aggregating object localization frames, namely multi-scale hierarchical clustering, is developed, which allows qualitative aggregation of localization frames from images with widely different scales.

[0014] The unique ability to perform target recognition directly on the thermal imaging module's processor gives the newly developed system a number of significant advantages over existing systems with external computing units. Summary of the invention

[0015] A group of inventions for "real-time detection of targets" allows for the implementation of intelligent image processing, target recognition algorithms on a processor within a thermal imaging module, and combines video recording and video content analysis in one device.

[0016] The result is a significantly improved processor performance and higher quality target recognition as part of an uncooled matrix thermal imaging module.

[0017] The technical effect is achieved by the fact that the system has a device for video recording, which is a thermal imaging module with a built-in computing unit based on a digital signal processor and includes a computing unit with a system-level chip, a RAM chip of one gigabit and a FLASH chip of 512 megabit (storage device used as a flash memory carrier), a power supply system composed of a pulse converter and an integrated power control circuit, which configures the required voltage for all nodes of the circuit board, a node for recording data to an SD drive (non-volatile memory card format), made in the form of an expansion card, mounted on a 60pin connector, where power, MMC signals (signals with a minimum frequency interval of "0" and "1") and UART come from the power board, the UART (Universal Asynchronous Receiver-Transmitter) is converted into RS-232 standard signals by a chip, the notable feature of which is the ability to provide conversion at a power supply voltage of 1.8V, an analog converter unit for amplifying and digitizing the video signal from the thermal imaging matrix, which is an input power supply stage composed of a linear regulator with a low voltage drop, and an Ethernet interface unit (a technological family for packet data transmission between computers and industrial network equipment).

[0018] The method for real-time data processing and target recognition in the computing unit of the thermal imaging module includes a basic signal processing unit, which includes error point replacement, seamless non-uniformity correction, noise suppression, thermal artifact suppression, dynamic range compression, optical distortion compensation, and then the processed signal transmission is compressed, recorded on a local storage medium and streamed, or frame processed by a recognition algorithm after the basic signal processing, which has a cascade structure consisting of a basic detector, a verification classifier and an artificial neural network (ANN). The person in the image is located by determining the coordinates of the frame describing the person in the image coordinate system. The basic element for locating the target in the image is a sliding window algorithm, which scans the image using a window of a given size. A scale pyramid generator is used to identify targets with a large range of size variations in the image. In order to ensure real-time operation, a method of generating HOG descriptors using a sliding window is used, and then the obtained descriptors are multiplied with the SVM classifier weight vector (reference vector method). When the final block normalization result is obtained, multiple HOG descriptors (special point descriptors, used for target recognition in computer vision and image processing) are generated simultaneously. In addition to the integer HOG descriptor, only one value can be stored in the memory, to which the result of the multiplication of the block and the fragment of the SVM weight vector is gradually added. The classified frames are processed using a multi-scale hierarchical clustering algorithm, aiming to extract the regions of interest, i.e., potential objects. The resulting clusters of objects of different sizes are formed using a multi-scale clustering technique, thereby generating overlapping sets of scales from all image scales, and hierarchical clustering is performed within each scale.

[0019] The resulting clusters with potential targets are then transferred to a validation classifier and an artificial neural network to verify the presence of targets in the region of interest and refine the location of the targets. The combination of a fast baseline detector and a high-quality validation algorithm makes it possible to achieve highly accurate and reliable real-time target recognition on the processor of the thermal imaging module. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The real-time target recognition system and method of the thermal imaging module is illustrated in the following schematic diagram:

[0021] Figure 1 A schematic diagram of the electronics housing showing the thermal imaging module.

[0022] Figure 2 A schematic diagram of the newly developed thermal imaging module is shown.

[0023] Figure 3 A schematic diagram of video data processing in the computing unit of the thermal imaging module is shown.

[0024] Figure 4 A schematic diagram showing the operation of the recognition algorithm.

[0025] Figure 5 A schematic diagram showing a single usage of a normalized HOG descriptor block.

[0026] Figure 6 A schematic diagram showing the step-by-step multiplication of vectors is shown.

[0027] Figure 7 An example of object recognition in thermal images is shown.

[0028] Figure numerals: 1 - lens, 2 - matrix detector, 3 - analog-digital unit, 4 - computing unit, 5 - power supply unit, 6 - Ethernet interface unit, 7 - electronic unit housing, 8 - thermal imaging matrix, 9 - memory, 10 - dynamic memory, 11 - flash memory, 12 - local storage medium (SD card), 13 - first window, 14 - second window, 15 - third window, 16 - first descriptor, 17 - second descriptor, 18 - third descriptor. DETAILED DESCRIPTION

[0029] As part of a thermal imaging module, the real-time data processing and target recognition system and method allows for intelligent image processing, target recognition algorithms to be implemented on a processor within the thermal imaging module, and combines video recording and video content analysis in one device ( Figure 1 ).

[0030] The newly developed video recording device is a thermal imaging module ( Figure 2 ), built-in digital signal processor ( Figure 3) includes a computing unit (4), a power system (5), an SD storage data recording unit (9, 10, 11), an analog converter unit (3) and an Ethernet interface unit (6). The power system (5) is composed of a pulse converter and an integrated power control circuit to configure the required voltage for all nodes of the circuit board. A 5V voltage is used as a nominal value.

[0031] The data logging unit (9, 10, 11) is designed as an expansion board mounted on a 60-pin connector. Power, MMC and UART signals come from the power board. UART is converted to RS-232 signals by a dedicated chip, the notable feature of which is that it can provide conversion at a power supply voltage of 1.8V.

[0032] The analog conversion unit (3) is used to amplify and digitize the video signal from the thermal imaging matrix. This unit is an input power supply stage consisting of a low voltage drop linear regulator. The analog conversion unit generates the voltages for the digital power supply and the analog power supply for the matrix, the analog-to-digital converter, and the matrix reference voltage source and the differential amplifier of the ADC (analog-to-digital converter). The reference voltage is generated by the digital-to-analog converter.

[0033] The physical layer of the MAC (a unique identifier assigned to each active device or some of its interfaces in an Ethernet computer network) and the PHY (an integrated circuit used to perform the physical layer functions of the network model) must be interoperable in order to transmit data over Gigabit Ethernet. For this purpose, a physical layer Ethernet transceiver was used in this development. The chip is powered by a two-step buck converter.

[0034] The computing node (4) includes a system-level chip, a 1-gigabit RAM chip and a 512-megabit FLASH memory chip.

[0035] Thanks to the built-in computing unit, the newly developed thermal imaging module is able to use its own computing resources to perform target recognition, video compression, data recording to local storage media and data transmission via a network interface, in addition to video recording and basic frame processing. Figure 3 ).

[0036] The newly developed target recognition method aims to locate targets of a given class in the image coordinate system in real time (25 frames per second) on a processor that is part of the thermal imaging module. Target localization in the image coordinate system means locating the target by selecting it in the image through the localization frame.

[0037] In order to reduce the consumption of computing resources, the developed recognition algorithm has a cascade structure consisting of a baseline detector, a verification classifier, and an artificial neural network (ANN). The unique combination of heterogeneous algorithms allows maintaining high quality of object recognition while significantly improving performance.

[0038] The person in the image is located by determining the coordinates of the frame describing the person in the image coordinate system. The basic element for locating an object in an image is the sliding window algorithm. This algorithm scans the image with a window of a given size. The scanning process starts from the upper left corner of the image, from where the first window is generated. The image fragments that fall into this window are analyzed for their similarity with the classification "human". The window is then moved to the left by a specified distance (sliding window step) and the analysis process is repeated. The movement process is repeated until the window reaches the right edge of the image, at which point the scanning bar moves down by a specified distance. Thus, the sliding window algorithm allows the entire image to be analyzed by moving stepwise from the upper left corner to the lower right corner.

[0039] The scale pyramid generator is used to identify objects with a wide range of sizes in an image. The original image scale is 640x480 pixels. The constant size of the scanning window is 32x64 pixels. Therefore, the first step of the scale pyramid searches for objects of approximately 30x60 pixels. In order to search for larger objects, the size of the original image is reduced by a specified amount (scaling step). Since the scanning window size remains unchanged, the ratio of the scanning window size to the image size increases, allowing the search for larger objects than in the first step. This image size reduction process is repeated until the image height is equal to the height of the scanning window, which corresponds to the search for objects with a height close to 480 pixels in the original image.

[0040] The classic sliding window algorithm used in the recognition algorithm generates a large number of windows, each of which needs to be analyzed. At the same time, the windows overlap with adjacent windows to a large extent, which means a lot of repeated calculations. In order to ensure real-time operation, the method of using sliding windows to generate HOG descriptors is adopted, and then the obtained descriptors are multiplied with the SVM classifier weight vector.

[0041] To optimize computation, a transition is made from pixel space to a precomputed grid of cells over which the sliding window will move. Each cell will then be generated once, avoiding a lot of repeated computation.

[0042] Since the minimum size of the sliding window step that provides reliable object recognition is equal to four pixels (at the first step of the scale pyramid), and the minimum step size for moving across the cell grid is equal to the size of one cell of the HOG descriptor, i.e. 8 pixels, the image is divided into 4x4 pixel fragments (sub-cells). This allows to eliminate repeated calculations without any change in the final recognition result.

[0043] Sub-cells are derived in a similar way to standard HOG descriptor cells and are 8-bin histograms. Next, the bins of four adjacent sub-cells must be added together to form a standard cell (32 additions). This eliminates the need for resource-intensive calculations of gradients, orientation angles, and histogram generation for each window. Therefore, at this step in the algorithm optimization, the HOG descriptor generation process is reduced to moving a sliding window over the sub-cell grid, adding the histograms of the four sub-cells to form a cell histogram, normalizing the HOG block histograms, concatenating the block histograms into the final descriptor and performing vector multiplication with the linear SVM classifier weights. There is no need to calculate the gradient and its direction pixel by pixel, nor to generate a cell orientation histogram for each window - these operations are performed once at each scale for the entire image.

[0044] However, repeated calculations that require a large number of operations also include the normalization of the HOG descriptor blocks.

[0045] In order to reduce the number of operations to normalize a block of HOG descriptors, it was decided to switch from a sliding window to a set of sliding blocks that move on a sub-cell grid. Thus, when a block is generated and normalized, the final representation of the block is placed as the concatenated normalized histogram at the corresponding positions of all HOG descriptors that make up the block ( Figure 5 ).

[0046] Therefore, when the final result of block normalization is obtained, multiple HOG descriptors are generated simultaneously. However, a significant disadvantage of this method is that a large number of generated HOG descriptors need to be constantly placed in RAM. The maximum number of descriptors that can be stored in RAM simultaneously using this method is eight. Since the size of a single HOG descriptor is 720x 16-bit elements, the size of the memory required is 11,520 bytes, which is a large portion of the L1 memory.

[0047] The result of using the HOG descriptor is a weighted sum, where the weights are the SVM classifier weights. Considering the nature of the weighted sum, it was decided to multiply the vectors part by part and then add the multiplication results to get the final result ( Figure 6 ).

[0048] Therefore, in addition to the integer HOG descriptor, only one value can be stored in memory, to which the multiplication results of the block and the SVM weight vector fragments are incrementally added.

[0049] The classification result of all the blocks that make up the HOG descriptor is the classified sliding window frame. The classified frame is processed using a multi-scale hierarchical clustering algorithm, aiming to extract the region of interest, i.e., the potential target. The hierarchical clustering algorithm generates the resulting clusters based on the input data, so that the target can be located more accurately. Since the position of the sliding window frame is fixed and depends on the sliding window step size and the scaling factor of the proportional pyramid generation algorithm, i.e., the number of changes in the position of the sliding window frame is limited, a more accurate positioning can be achieved. In turn, the position of the resulting cluster is calculated based on the coordinates of the sliding window frames that form the cluster and the classification results of these frames. Therefore, the position of the resulting cluster is not limited to a limited set of possible positions, but is calculated based on the obtained target position data.

[0050] The resulting clusters of objects of different sizes are formed using a multi-scale clustering technique, thereby generating overlapping sets of scales from all image scales and performing hierarchical clustering within each scale set. This technique is able to successfully generate resulting clusters of objects across the entire range of sizes. The division of scales into overlapping sets is done based on the results of empirical data obtained from testing.

[0051] The resulting clusters with potential targets are then transferred to a validation classifier and an artificial neural network (ANN) to verify the presence of targets in the region of interest and refine their location. The combination of a fast baseline detector and a high-quality validation algorithm makes it possible to combine high accuracy and reliability of target recognition that are no less than modern ANN architectures with significantly lower computational complexity required for real-time operation on the thermal imaging module processor compared to modern ANN architectures.

[0052] Examples of the results of the recognition algorithm as part of the thermal imaging module are shown in Figure 7 .

[0053] The unique ability to perform target recognition directly on the thermal imaging module processor gives the newly developed system a number of significant advantages over existing systems with external computing units.

[0054] Implementing the recognition algorithm on the thermal imaging module processor reduces the latency of video stream processing to a minimum, since there is no need to transmit the video stream to a peripheral device. This is very important when using recognition algorithms in a strict real-time system. In addition, not having to transmit the video stream significantly reduces the load on the system communication channel.

[0055] Another significant advantage of integrating the newly developed thermal imaging modules is the high scalability and reconfigurability of the system. Scalability is ensured by the fact that as the number of thermal imaging modules in the system increases, the computing power of the system and the bandwidth of the communication channel do not need to increase accordingly, because all necessary processing is done directly by the camera itself. Reconfigurability is achieved by the fact that the integration of thermal imaging modules does not require a dedicated computing unit or a broadband communication link for video streaming.

Claims

1. The real-time target recognition system includes a video recording device, which is a thermal imaging module with a built-in digital signal processor-based computing unit, characterized in that: It contains the following components: a computing unit with a system-on-chip, a 1-gigabit RAM chip, and a 512-megabit FLASH chip (a storage device used as a flash memory carrier); A power supply system, which includes a pulse converter and an integrated power control circuit, which configures the necessary voltages for all nodes of the circuit board; a node for recording data to an SD drive, made in the form of an expansion card, mounted on a 60pin connector, where the power supply, MMC signal and UART come from the power board, and the UART is converted into RS-232 standard signals by a chip, whose notable feature is the ability to provide conversion at a power supply voltage of 1.8V; an analog converter unit for amplifying and digitizing the video signal from the thermal imaging matrix, which is an input power supply stage of a linear regulator with a low voltage drop; and an Ethernet interface unit.

2. A method for real-time target recognition using the system of claim 1, comprising the following steps: The basic signal processing steps performed in the basic signal processing unit include fault point replacement, seamless non-uniformity correction, noise suppression, thermal artifact suppression, dynamic range compression, and optical distortion compensation; data processing using a compression algorithm in the computing unit of the thermal imaging module, where the video stream is recorded to a local storage medium and transmitted, or the computing unit of the thermal imaging module uses a recognition algorithm with a cascade structure consisting of a baseline detector, a verification classifier, and an artificial neural network; the baseline detector processes the positioning of a person in the image by determining the coordinates of the frame describing the person in the image coordinate system, the sliding window algorithm scans the image with a window of a given size to determine the location of the target, the scale pyramid generator recognizes the variable image, and the HOG descriptor is generated using a sliding window, and then the obtained descriptor is multiplied with the SVM classifier weight vector to provide real-time recognition.

3. The method for real-time target recognition according to claim 2, characterized in that: The baseline detector is used in the computational unit of the thermal imaging module, where multiple HOG descriptors are formed simultaneously to normalize the patches.

4. The method for real-time target recognition according to claim 2, characterized in that: The computing unit of the thermal imaging module has a baseline detector, wherein a value can be stored in a memory, to which value the results of the multiplication of the blocks with the segments of the SVM weight vector are gradually added.

5. The method for real-time target recognition according to claim 2, characterized in that: The computing unit of the thermal imaging module processes the classified frames using a recognition algorithm including a validation classifier, aiming to select regions of interest to form resulting clusters of objects of different sizes using multiscale clustering, by which overlapping scale sets are generated from all image scales, and hierarchical clustering is performed within each scale set.

6. The method for real-time target recognition according to claim 2, characterized in that: The computing unit of the thermal imaging module uses a recognition algorithm containing an artificial neural network, which checks the presence of a target in the area of ​​interest and refines the target's location.

Images