Audio and video stream real-time face recognition intelligent tracking and labeling system

Abstract

The invention relates to the technical field of face recognition, in particular to an intelligent tracking and labeling system for real-time face recognition of audio and video streams, which comprises a data acquisition and analysis module for synchronously acquiring a visible light video stream and an infrared multi-mode sensing data stream of a target area, and acquiring a real-time face recognition data stream from the infrared multi-mode sensing data stream; analyzing two dynamic physiological signals related to the target life activity; the feature fusion module is used for fusing and generating a physiological feature identifier of a target based on the time sequence change modes and the spatial distribution features of the two dynamic physiological signals, extracting the time sequence rhythm features and the spatial distribution features of the two dynamic physiological signals, and fusing the time sequence rhythm features and the spatial distribution features to generate the physiological feature identifier; through real-time binding of the physiological feature identifier and the heat source space position in the infrared thermal imaging data, continuous positioning and identity association can be performed according to the physiological features of the target in the motion process of the target, and continuous face tracking is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of face recognition, and particularly relates to an audio and video stream real-time face recognition intelligent tracking and labeling system. BACKGROUND

[0002] With the rapid development of public safety, intelligent security and human-computer interaction fields, higher requirements are put forward for real-time, non-contact identification and continuous tracking technology of personnel identity. Traditional face recognition technology mainly relies on facial texture and geometric features collected by a camera, and although the accuracy is high, it still faces many inherent limitations in practical application. First, its performance is severely restricted by environmental lighting conditions, and the recognition rate significantly decreases in dim, backlight or strong light direct scene. Second, face occlusion can cause key feature loss, resulting in recognition failure. At the same time, existing personnel tracking is mostly based on pure visual tracking algorithm, and once the target is temporarily lost or the appearance changes greatly, it is easy to cause tracking interruption or identity confusion. SUMMARY

[0003] The present application provides an audio and video stream real-time face recognition intelligent tracking and labeling system to solve the technical problems in the prior art.

[0004] The technical solution of the present application to solve the above technical problems is as follows: an audio and video stream real-time face recognition intelligent tracking and labeling system, comprising: a data acquisition and analysis module: synchronously acquiring a visible light video stream and an infrared multi-modal sensing data stream of a target area, and analyzing two dynamic physiological signals related to target life activities from the infrared multi-modal sensing data stream; a feature fusion module: based on the time sequence change mode and spatial distribution features of the two dynamic physiological signals, fusing to generate a physiological feature identifier of the target, extracting the time sequence rhythm features and spatial distribution features of the two dynamic physiological signals, fusing the time sequence rhythm features and spatial distribution features to generate a physiological feature identifier; an identity recognition module: performing identity recognition based on the physiological feature identifier, and determining a target motion trajectory in real time based on the spatial change of the infrared multi-modal sensing data stream; an identity labeling module: labeling the target in the visible light video stream based on the recognition and tracking results.

[0005] In a preferred embodiment, the data acquisition and analysis module synchronously acquires a visible light video stream and an infrared multi-modal sensing data stream of the target region, the infrared multi-modal sensing data stream specifically includes a thermal imaging data stream generated by an infrared thermal imaging sensor, the thermal imaging data stream reflects the surface temperature distribution of each target in the scene in the form of an image sequence, and an infrared spectrum data stream generated by an infrared spectrum sensor, the infrared spectrum data stream contains spectral intensity information of a specific infrared waveband; From the infrared multi-modal sensing data stream, a dynamic physiological signal related to the target life activity is parsed, the target region is locked through image processing technology, each frame of image in the thermal imaging data stream is analyzed, a preset temperature range threshold is set, the preset temperature range threshold is set as a typical human body surface temperature interval higher than the ambient background temperature, all continuous pixel regions with temperature values falling within the preset temperature range threshold are identified in the image, and the continuous regions are preliminarily marked as potential human target candidate regions, the region most consistent with the typical thermal distribution characteristics is selected from the human target candidate regions according to the typical thermal distribution characteristics of the face region in the thermal imaging, and the region is locked as the target region; After successfully locking the target region, the temperature values of all pixel points in the locked region are continuously extracted from each subsequent frame of synchronously acquired thermal imaging data, the average temperature of the target region in each frame is calculated as an instantaneous temperature observation value, the instantaneous temperature observation values are continuously recorded at a fixed time interval to form a temperature value sequence continuously changing with time; From the temperature value sequence, all marked peak points and valley points are identified, and the peak points and valley points are respectively classified into a peak sequence and a valley sequence in time sequence, each peak point represents the vertex of a temperature rising trend, and each valley point represents the bottom point of a temperature falling trend, the peak sequence and the valley sequence are processed respectively, for the peak sequence, the time interval between each two adjacent peak points is calculated to obtain a peak interval value, and for the valley sequence, the time interval between each two adjacent valley points is calculated to obtain a valley interval value; Based on all the calculated interval values, it is judged whether the interval values fall within a preset typical time range consistent with the human respiratory rhythm, it is checked whether the difference between the longest interval value and the shortest interval value is less than a maximum allowed fluctuation range threshold, if most of the interval values are within the typical time range, and the maximum difference between the longest interval value and the shortest interval value does not exceed the maximum allowed fluctuation range threshold, it is determined that the fluctuations have regularity; If the regularity condition is met, the temperature fluctuation is confirmed as a periodic fluctuation related to respiratory activity, the average value of all interval values is calculated as the average fluctuation period, the average amplitude of the peak and the trough is recorded as the fluctuation intensity, the average fluctuation period and the fluctuation intensity are taken as the rhythm characteristics of the periodic fluctuation, and the rhythm characteristics are taken as the respiratory thermal disturbance signal reflecting the target respiratory rhythm; Based on the target region, the boundary coordinates of the target region in the thermal imaging data are obtained, and the spatial coordinate mapping relationship is used. Based on the pre-stored spatial calibration parameters of the infrared thermal imaging sensor and the infrared spectrum sensor, the target region center point pixel coordinates locked in the thermal imaging data stream are first converted into a unit observation direction vector in the thermal imaging sensor coordinate system. The unit observation direction vector is converted to the infrared spectrum sensor coordinate system through coordinate conversion. According to the type of the infrared spectrum sensor, the direction vector is converted into the instruction parameters required by the infrared spectrum sensor to determine the target corresponding detection region for analysis in the infrared spectrum data. According to the characteristic absorption wavelength of the specific component to be detected in the target exhaled gas, a plurality of wave bands contained in the infrared spectrum data stream are selected as core observation wave bands for analyzing the gas concentration change. In the continuously collected infrared spectrum data stream, the signal intensity values of the selected characteristic absorption wavelength adjacent wave bands are extracted in the target corresponding detection region in real time, and the signal intensity values are sequentially recorded according to the time stamp synchronized with the thermal imaging data to form a characteristic wave band intensity original sequence. The characteristic wave band intensity original sequence is preprocessed, and the preprocessing includes baseline correction, trend removal and noise suppression operation. After the preprocessing of the characteristic wave band intensity original sequence, a corrected intensity sequence is obtained. The corrected intensity sequence is analyzed to detect the periodic fluctuation component existing therein, and the periodic fluctuation component is matched and verified with the period of the respiratory thermal disturbance signal analyzed from the thermal imaging data stream. When the two periods are matched, the periodic intensity fluctuation component that has passed the period matching verification is extracted as a time-series spectrum fluctuation signal reflecting the gas exchange dynamics. The two kinds of dynamic physiological signals, the respiratory thermal disturbance signal and the time-series spectrum fluctuation signal, are output.

[0006] In a preferred embodiment, the feature fusion module extracts features from the two kinds of dynamic physiological signals. For the respiratory thermal disturbance signal analyzed from the thermal imaging data stream, the time sequence features reflecting the respiratory rhythm are extracted, including periodic fluctuation, period value, amplitude value and stability parameter. The periodic fluctuation is identified by analyzing the temperature value sequence of the respiratory thermal disturbance signal, identifying the repeatedly occurring rising and falling waveforms with an amplitude exceeding a preset minimum threshold, and defining each waveform as an independent respiratory cycle. The cycle value is obtained by measuring the duration of each identified respiratory cycle. The amplitude value is obtained by measuring the difference between the peak and valley values of the temperature fluctuation in each identified respiratory cycle. The stability parameter is calculated based on the statistical distribution indicators of the calculated respiratory cycle value and the respiratory amplitude value, including the coefficient of variation of the cycle value and the standard deviation of the amplitude value, and the statistical distribution indicators are used as parameters representing the stability of the respiratory rhythm. Meanwhile, the spatial heat field distribution characteristics corresponding to the respiratory thermal disturbance signal are extracted, including core spatial position positioning and spatial diffusion range determination. The core spatial position positioning is achieved by determining the continuous pixel region with the most significant intensity of the respiratory thermal disturbance signal in the image of the thermal imaging data stream, and calculating the pixel coordinates of the center point of the continuous pixel region as the core spatial position. The spatial diffusion range determination is performed by expanding outward from the core spatial position until reaching the boundary pixels where the temperature fluctuation amplitude decays to below a preset proportion threshold. The region surrounded by these boundary pixels is defined as the spatial diffusion range. For the time-series spectral fluctuation signal parsed from the spectral data stream, its time-series characteristics reflecting gas exchange dynamics are extracted, including periodic fluctuation identification, fluctuation period, and peak-to-trough intensity ratio. The periodic fluctuation identification is achieved by analyzing the intensity value sequence of the time-series spectral fluctuation signal, identifying the periodically varying waveforms corresponding to exhalation and inhalation processes that repeatedly occur. The fluctuation period is obtained by measuring the time interval between adjacent points with the same phase. The peak-to-trough intensity ratio is calculated by reading the peak intensity value and the trough intensity value within a fluctuation period, and then calculating the ratio of the peak intensity value to the trough intensity value. Meanwhile, its corresponding spatial characteristics are extracted, including determining the detection region position identifier and converting it to standard spatial coordinates. The detection region position identifier is determined by directly reading the unique identifier of the target corresponding detection region in the spectral sensor when generating the time-series spectral fluctuation signal. The unique identifier is converted to a set of reference coordinates in the thermal imaging field coordinate system. After completing the feature extraction, the time-series characteristic sequence of the respiratory thermal disturbance signal and the time-series characteristic sequence of the spectral fluctuation signal are aligned on the time axis to ensure that the data points representing the same physiological event in the two signal sequences are completely matched in time, forming a time-aligned time-series rhythm characteristic combination. The time-aligned timing rhythm features are combined, and the spatial distribution features extracted from the homologous signals are spliced, the timing rhythm features of the respiratory heat disturbance signal are connected at the head and tail with the spatial heat field distribution features thereof, the timing rhythm features of the spectral fluctuation signal are connected at the head and tail with the spatial position features thereof, and the two groups of connected features are spliced again as a long feature vector; After splicing, the long feature vector formed by splicing is weighted, the feature values in different dimensions in the long feature vector are scaled based on preset weight coefficients, the preset weight coefficients are set based on the contribution of each feature to distinguishing different individuals in historical data, and the weighted composite feature vector is taken as a target physiological feature identifier; In a preferred embodiment, during continuous monitoring of the target area, when the physiological feature identifiers collected and generated in a continuous time period exhibit high consistency, a unique identity identifier is assigned to the target. The heat source area corresponding to the current identified target is located from the infrared multi-modal sensing data stream, the instantaneous spatial position coordinates are determined, the instantaneous spatial position coordinates acquired in time sequence are connected, and a continuous trajectory reflecting the moving path of the target is generated.

[0007] In a preferred embodiment, the identity labeling module takes the identity identifier and the continuous trajectory of the target moving path as input, the continuous trajectory is arranged in time sequence, reflects the spatial position sequence of the continuous moving path of the target, and the identity identifier and the spatial position sequence are synchronously acquired together with the current visible light video stream frame to be processed. The visible light video stream frame is aligned with the time point of the identity identifier and the spatial position sequence, the real-time position of the target associated with the spatial position coordinates and the identity identifier in the continuous trajectory data is mapped to the corresponding pixel coordinates of the current visible light video frame, the pixel coordinates obtained by mapping are used to generate a labeling graphical element including the identity identifier and a trajectory line connecting historical position points, the labeling graphical element is rendered to the corresponding coordinate position in real time through image superposition technology, a labeling picture is formed, and the trajectory line is continuously updated in the subsequent frames of the video stream to draw the latest moving path of the target.

[0008] The present application has the following advantages: by fusing the respiratory heat disturbance signal derived from infrared thermal imaging and the spectral fluctuation signal derived from infrared spectrum, the physiological feature identifier is constructed, the physiological feature identifier is real-time bound with the heat source spatial position in the infrared thermal imaging data, continuous positioning and identity association are realized according to the physiological features of the target during target motion, continuous face tracking is realized, the physiological identity information and the spatial trajectory information obtained by the infrared sensing domain are synchronized in time and space and mapped in coordinates, and are superimposed into the visible light video stream, thereby providing a visual interface displaying the motion trajectory. Attached Figure Description

[0009] Figure 1 This is a system block diagram of the present invention; Figure 2 This is a logic flowchart of the present invention. Detailed Implementation

[0010] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0012] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0013] like Figures 1-2 This embodiment provides: a real-time face recognition intelligent tracking and annotation system for audio and video streams, comprising: Data acquisition and analysis module: synchronously acquires visible light video stream and infrared multimodal sensing data stream of the target area, and extracts two dynamic physiological signals related to the target's life activities from the infrared multimodal sensing data stream; In this embodiment of the invention, the data acquisition and analysis module needs to be specifically described. The data acquisition and analysis module simultaneously acquires the visible light video stream and the infrared multimodal sensing data stream of the target area. The infrared multimodal sensing data stream specifically includes the thermal imaging data stream generated by the infrared thermal imaging sensor, which reflects the surface temperature distribution of each target in the scene in the form of an image sequence, and the infrared spectral data stream generated by the infrared spectral sensor, which contains spectral intensity information of a specific infrared band. Dynamic physiological signals related to the target's life activities are extracted from the infrared multimodal sensing data stream. The target area is located using image processing technology. Each frame of the thermal imaging data stream is analyzed, and a preset temperature range threshold is set. The preset temperature range threshold is set to a typical human body surface temperature range (e.g., 30℃-37℃) that is higher than the ambient background temperature. Continuous pixel areas in the image with all temperature values ​​falling within the preset temperature range threshold are identified and preliminarily marked as potential human target candidate areas. Based on the typical thermal distribution characteristics of the face area in thermal imaging (e.g., the nasal cavity and oral cavity openings usually show local high temperature points, and these high points have a specific relative positional relationship in the facial area), the area that best matches the typical thermal distribution characteristics is selected from the human target candidate areas and locked as the target area (e.g., the facial area). After successfully locking the target area, the temperature values ​​of all pixels in the locked area are continuously extracted from the thermal imaging data synchronously acquired in each subsequent frame. The average temperature of the target area in each frame is calculated as an instantaneous temperature observation value. The instantaneous temperature observation values ​​are continuously recorded at fixed time intervals (such as the frame rate of the video stream) to form a temperature value sequence that changes continuously over time. Identify all marked peaks and troughs in the temperature value sequence, and classify them into peak sequences and trough sequences according to the time series. Each peak represents the apex of a temperature rise trend, and each trough represents the trough of a temperature fall trend. Process the peak sequences and trough sequences separately. For the peak sequence, calculate the time interval between every two adjacent peaks to obtain the peak interval value. For the trough sequence, calculate the time interval between every two adjacent troughs to obtain the trough interval value. Based on all the calculated interval values, it is determined whether these interval values ​​fall within a preset typical time range that conforms to the human breathing rhythm, for example, corresponding to a cycle range of 12-20 breaths per minute. It is checked whether the difference between the longest interval value and the shortest interval value is less than the maximum allowable fluctuation range threshold. If most interval values ​​are within the typical time range and the maximum difference between the longest interval value and the shortest interval value does not exceed the maximum allowable fluctuation range threshold, then these fluctuations are determined to be regular. If the regularity condition is met, the temperature fluctuation is confirmed as a periodic fluctuation related to respiratory activity, the average value of all interval values is calculated as the average fluctuation period, the average amplitude of the peak and the trough is recorded as the fluctuation intensity, the average fluctuation period and the fluctuation intensity are taken as the rhythm characteristics of the periodic fluctuation, and the rhythm characteristics are taken as the respiratory thermal disturbance signal reflecting the target respiratory rhythm; Based on the target region, the boundary coordinates of the target region in the thermal imaging data are obtained, and the spatial coordinate mapping relationship is used to convert the target region center point pixel coordinates locked in the thermal imaging data stream into a unit observation direction vector in the thermal imaging sensor coordinate system based on the pre-stored spatial calibration parameters of the infrared thermal imaging sensor and the infrared spectrum sensor. The unit observation direction vector is converted to the infrared spectrum sensor coordinate system through coordinate conversion. According to the type (point scanning or imaging) of the infrared spectrum sensor, the direction vector is converted into the instruction parameters required by the infrared spectrum sensor to determine the corresponding detection region of the target in the infrared spectrum data for analysis. It should be noted that the instruction parameters required by the infrared spectrum sensor are control instructions for driving the infrared spectrum sensor to move, which directly determine the final pointing of the sensor detection beam. According to the type of infrared spectrum sensor, the instruction parameters include: angle instruction: if a rotating holder is used, the parameters are horizontal and vertical angles; position coordinates: if a two-dimensional translation table is used, the parameters are X-axis and Y-axis coordinate positions; voltage or digital signal: if a galvanometer is used, the parameters are analog voltages or digital code values for controlling the deflection of the mirror. These parameters are calculated according to the unit observation direction vector converted from the thermal imaging coordinate to the spectrum sensor coordinate system; It should be noted that the spatial coordinate mapping relationship is obtained by placing a reference calibration object with a known spatial position in the target region during system deployment or calibration, synchronously collecting and recording its accurate pixel coordinates in the thermal imaging sensor image and the corresponding position identifier in the infrared spectrum sensor, repeatedly collecting multiple sets of paired data by changing positions, forming an initial sample set, associating the multiple sets of paired data one by one based on the sample set, dividing the thermal imaging field of view into continuous areas according to the calibration object, determining the most common spectrum detection position identifier for each area, and generating a lookup table that maps each thermal imaging pixel unit to a spectrum detection identifier, which is used as a fixed mapping rule. After generation, the calibration data is used for verification and the areas with large errors are marked. In actual operation, the system directly applies this mapping rule to convert the target pixel coordinates locked in the thermal imaging to the spectrum detection position identifier to determine the corresponding detection region.

[0014] According to the characteristic absorption wavelength of the specific component (e.g. carbon dioxide) to be detected in the target exhaled gas, a plurality of wave bands contained in the infrared spectrum data stream are selected as core observation wave bands for analyzing the gas concentration change, which are adjacent to the characteristic absorption wavelength; In the continuously collected infrared spectrum data stream, the signal intensity values of the selected wave bands adjacent to the characteristic absorption wavelength are extracted in real time within the target corresponding detection area, and the signal intensity values are sequentially recorded according to the time stamp synchronized with the thermal imaging data to form a characteristic wave band intensity original sequence; The characteristic wave band intensity original sequence is pre-processed, and the pre-processing includes baseline correction, trend removal and noise suppression operation. After the pre-processing of the characteristic wave band intensity original sequence, a corrected intensity sequence is obtained; The corrected intensity sequence is analyzed to detect the periodic fluctuation component existing therein. The periodic fluctuation component is that the signal intensity decreases due to the increase of the concentration of a specific gas component during exhalation, and the opposite is true during inhalation. Therefore, the respiratory activity will produce a periodic fluctuation on the corrected intensity sequence. The periodic fluctuation component is matched and verified with the period of the respiratory thermal disturbance signal analyzed from the thermal imaging data stream. When the two periods are matched, the periodic intensity fluctuation component that has passed the period matching verification is extracted as a time-series spectrum fluctuation signal reflecting the gas exchange dynamics; It should be noted that the pre-stored spatial calibration parameters of the infrared thermal imaging sensor and the infrared spectrum sensor are obtained through a pre-calibration process, and at least include the following information: Infrared thermal imaging sensor parameters: such as focal length, principal point coordinates, distortion coefficient, used for converting thermal imaging pixel coordinates to standardized observation direction with sensor optical center as origin: Infrared spectrum sensor parameters: including focal length, principal point and other parameters; External parameters: rotation matrix and translation vector describing the relative position and attitude between the thermal imaging sensor coordinate system and the infrared spectrum sensor coordinate system, which defines how the direction vector observed from one sensor is converted to the coordinate system of another sensor.

[0015] The two dynamic physiological signals, the respiratory thermal disturbance signal and the time-series spectrum fluctuation signal, are output. The two dynamic physiological signals are respectively from the physical thermal radiation and the gas chemical component absorption, and jointly represent the respiratory life activity of the same individual.

[0016] It should be noted that the maximum allowable fluctuation range threshold is the maximum time difference allowed to exist between consecutive breathing cycles when analyzing the respiratory thermal disturbance signal. Although normal breathing has rhythm, its period is not completely fixed and there is a slight natural fluctuation. The maximum allowable fluctuation range threshold is to distinguish the normal physiological fluctuation from the irregular abnormal fluctuation that may be caused by noise or interference. If the time difference between the calculated multiple breathing cycles is less than the maximum allowable fluctuation range threshold, it is considered that the breathing rhythm is stable and regular, otherwise, it is considered that the rhythm is disordered, and the signal does not belong to regular breathing activity. The maximum allowable fluctuation range threshold is a process of combining physiological knowledge and scene adaptive fine-tuning. The maximum allowable fluctuation range threshold is preset according to the statistical characteristics of human respiratory physiology. The normal range of breathing cycle fluctuation is determined by statistical analysis of a large number of healthy individuals in a resting state.

[0017] The feature fusion module fuses the physiological feature identifier of the target based on the time sequence change mode and the spatial distribution feature of the two dynamic physiological signals, extracts the time sequence rhythm feature and the spatial distribution feature of the two dynamic physiological signals, fuses the time sequence rhythm feature and the spatial distribution feature, and generates the physiological feature identifier. In the embodiment of the application, it should be specifically noted that the feature fusion module extracts features from two dynamic physiological signals. For the respiratory thermal disturbance signal parsed from the thermal imaging data stream, the time sequence features reflecting the respiratory rhythm are extracted, including periodic fluctuation, period value, amplitude value and stability parameter. The periodic fluctuation is analyzed by analyzing the temperature value sequence of the respiratory thermal disturbance signal, identifying the rising and falling waveforms with an amplitude exceeding a preset minimum threshold that repeatedly appears, and defining each waveform as an independent breathing cycle. The period value is obtained by measuring the duration of each identified breathing cycle, i.e. the time length from the starting point of a waveform to the starting point of the next adjacent waveform. The amplitude value is obtained by measuring the difference between the peak value and the valley value of the temperature fluctuation in each identified breathing cycle. The stability parameter is calculated based on the calculated respiratory period value and respiratory amplitude value, and the statistical distribution indexes of them are calculated, including the coefficient of variation of the period value and the standard deviation of the amplitude value. The statistical distribution indexes are used as parameters representing the stability of the respiratory rhythm. It should be noted that the preset minimum threshold is a critical value for distinguishing the temperature fluctuation caused by real breathing from background noise. The data is collected: under the target monitoring distance and environmental conditions, a thermal imaging sequence containing still and breathing individuals is collected, and the real breathing rhythm is recorded as a reference. In the determined breathing area, the temperature fluctuation amplitude range caused by real breathing is measured, and in the background area without breathing activity or non-living body area, the temperature fluctuation amplitude is measured as the environmental and sensor noise level. The preset minimum threshold is usually set to be higher than the amplitude of noise fluctuation, while ensuring that it covers the minimum value of most real breathing fluctuations.

[0018] Meanwhile, the spatial heat field distribution characteristics corresponding to the respiratory heat disturbance signal are extracted, including core spatial position positioning and spatial diffusion range determination. The core spatial position positioning is determined by determining the continuous pixel area with the most significant respiratory heat disturbance signal intensity in the image of the thermal imaging data stream, such as the maximum average temperature fluctuation amplitude, calculating the pixel coordinates of the center point of the continuous pixel area as the core spatial position. The spatial diffusion range determination is determined with the core spatial position as the center and extended outward until the boundary pixels where the temperature fluctuation amplitude decays to below the preset proportion threshold. The area surrounded by these boundary pixels is defined as the spatial diffusion range. It should be noted that the preset proportion threshold is a quantitative standard for defining the effective heat disturbance space directly related to the target physiological activity. When determining the spatial diffusion range of the respiratory heat disturbance signal, search outward from the core position with the strongest signal, and determine the point where the temperature fluctuation amplitude drops below a certain percentage of the core position fluctuation amplitude as the boundary of the effective area. If the preset proportion threshold is 50% and the core position respiratory fluctuation amplitude is 2℃, it will be extended outward until the pixel point where the respiratory fluctuation amplitude decays to below 1℃ is found. The boundary formed by connecting these pixel points is taken as the spatial diffusion range.

[0019] For the time-series spectral fluctuation signal parsed from the spectral data stream, its time-series features reflecting gas exchange dynamics are extracted, including periodic fluctuation identification, fluctuation period, and peak-to-trough intensity ratio. The periodic fluctuation identification is identified by analyzing the intensity value sequence of the time-series spectral fluctuation signal, identifying the periodically varying waveform corresponding to the exhalation and inhalation process that repeatedly appears in the sequence. The fluctuation period is obtained by measuring the time interval between two adjacent points of the same phase (such as two consecutive troughs). The peak-to-trough intensity ratio is calculated by reading the peak intensity value and the trough intensity value within a fluctuation period, and calculating the ratio of the peak intensity value to the trough intensity value. Meanwhile, the corresponding spatial features are extracted, the spatial features including determining a detection area position identifier and converting the detection area position identifier into standard spatial coordinates, the detection area position identifier being determined by directly reading a unique identifier of a target corresponding detection area in a spectral sensor when generating the time-series spectral fluctuation signal, and converting the unique identifier into a set of reference coordinates in a thermal imaging field of view coordinate system; After the feature extraction is completed, the time-series feature sequence of the respiratory thermal disturbance signal and the time-series feature sequence of the spectral fluctuation signal are aligned on a time axis to ensure that data points representing the same physiological event in the two signal sequences are completely matched in time, forming a time-aligned time-series rhythm feature combination; The time-aligned time-series rhythm feature combination is spliced with the spatial distribution features extracted from the homologous signal, the time-series rhythm features of the respiratory thermal disturbance signal and the spatial thermal field distribution features thereof are connected at the head and tail, and the time-series rhythm features of the spectral fluctuation signal and the spatial position features thereof are connected at the head and tail, and the two groups of features after the connection are spliced again to form a long feature vector; After the splicing is completed, the long feature vector formed by the splicing is weighted, the feature values in different dimensions in the long feature vector are scaled based on preset weight coefficients, the preset weight coefficients are set based on the contribution of each feature to distinguishing different individuals in historical data, and the weighted composite feature vector is taken as a physiological feature identifier of the target, which represents the unique representation of the target individual on the respiratory thermal radiation and the spectral features of the exhaled gas. The identity recognition module is configured to perform identity recognition based on the physiological feature identifier and determine a target motion trajectory in real time based on spatial changes of the infrared multi-modal sensing data stream. In the embodiment of the present application, it needs to be specifically explained that the identity recognition module, in the continuous monitoring process of the target area, when the physiological feature identifiers collected and generated in a continuous time period show high consistency, a unique identity identifier is assigned to the target, thereby completing the identity differentiation and identification of a specific target. The heat source area corresponding to the currently identified target is located from the infrared multi-modal sensing data stream, the instantaneous spatial position coordinates are determined, the instantaneous spatial position coordinates acquired in time sequence are connected to generate a continuous trajectory reflecting the movement path of the target.

[0020] The identity annotation module is configured to annotate the target in the visible light video stream based on the recognition and tracking results. In the embodiment of the present application, it needs to be particularly pointed out that the identity labeling module, the identity identifier and the continuous track of the target moving path are taken as inputs, the continuous track is arranged in time sequence, reflects the spatial position sequence of the continuous moving path of the target, and the identity identifier and the spatial position sequence and the current visible light video stream frame to be processed are synchronously acquired; The visible light video stream frame is aligned with the time point of the identity identifier and the spatial position sequence, the real-time position of the target associated with the spatial position coordinates and the identity identifier in the continuous track data is mapped into the corresponding pixel coordinates of the current visible light video frame, the pixel coordinates obtained through the mapping are used to generate a labeling graphic element, including the identity identifier and the track line connecting the historical position points, the labeling graphic element is rendered to the corresponding coordinate position in real time through the image superposition technology, a labeling picture is formed, and the track line is continuously updated in the subsequent frames of the video stream to draw the latest moving path of the target.

[0021] It needs to be pointed out that in the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0022] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media containing computer usable program codes (including but not limited to disk memory, CD-ROM, optical memory, etc.).

[0023] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded computer or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0024] These computer program instructions can also be stored in a computer readable memory capable of guiding the computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer readable memory produce a product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams.Figure 1 one or more processes and / or functions specified in the block or blocks. Figure 1

[0025] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable data processing devices provide a process for implementing the functions specified in the flowchart Figure 1 one or more processes and / or functions specified in the block or blocks. Figure 1

[0026] Although preferred embodiments of the application have been described, those skilled in the art will recognize that additional modifications and variations may be made thereto without departing from the spirit and scope of the application. It is therefore intended that the appended claims cover all such modifications and variations as fall within the scope of the application.

[0027] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore intended that the invention be covered within the scope of the appended claims, and their equivalents.​​

Claims

1. An audio and video stream real-time face recognition intelligent tracking and labeling system, characterized in that, The method comprises the following steps: a data acquisition and analysis module: synchronously acquiring a visible light video stream and an infrared multi-modal sensing data stream of a target region, and analyzing two dynamic physiological signals related to target life activities from the infrared multi-modal sensing data stream; a feature fusion module: fusing physiological feature identifiers of the target based on the time sequence change mode and spatial distribution characteristics of the two dynamic physiological signals, extracting time sequence rhythm characteristics and spatial distribution characteristics of the two dynamic physiological signals, fusing the time sequence rhythm characteristics and the spatial distribution characteristics, and generating the physiological feature identifiers; an identity recognition module: performing identity recognition based on the physiological feature identifiers, and determining a target motion trajectory in real time based on spatial changes of the infrared multi-modal sensing data stream; an identity labeling module: labeling the target in the visible light video stream based on the recognition and tracking results. 2.The audio-video stream real-time face recognition intelligent tracking and labeling system according to claim 1, characterized in that, The data acquisition and analysis module synchronously acquires a visible light video stream and an infrared multi-modal sensing data stream of a target region, wherein the infrared multi-modal sensing data stream specifically comprises a thermal imaging data stream generated by an infrared thermal imaging sensor, and the thermal imaging data stream reflects surface temperature distribution of each target in a scene in the form of an image sequence, and an infrared spectrum data stream generated by an infrared spectrum sensor, and the infrared spectrum data stream contains spectral intensity information of a specific infrared waveband. The two dynamic physiological signals related to target life activities are analyzed from the infrared multi-modal sensing data stream, the target region is locked through image processing technology, each frame of image in the thermal imaging data stream is analyzed, a preset temperature range threshold is set, the preset temperature range threshold is set as a typical human body surface temperature interval higher than an environmental background temperature, all continuous pixel regions with temperature values falling within the preset temperature range threshold are identified in the image, and the continuous regions are preliminarily marked as potential human target candidate regions, a region most consistent with the typical thermal distribution characteristics is selected from the human target candidate regions according to the typical thermal distribution characteristics of the face region in the thermal imaging, and the region is locked as the target region. 3.The audio-video stream real-time face recognition and intelligent tracking and labeling system according to claim 2, characterized in that, After successfully locking the target region, the temperature values of all pixel points in the locked region are continuously extracted from each frame of the subsequently synchronously acquired thermal imaging data, the average temperature of the target region in each frame is calculated as an instantaneous temperature observation value, the instantaneous temperature observation values are continuously recorded at a fixed time interval, and a temperature value sequence continuously changing with time is formed. All marked wave peak points and wave trough points are identified from the temperature value sequence, and the wave peak points and the wave trough points are respectively classified into a wave peak sequence and a wave trough sequence in time sequence, each wave peak point represents a vertex of a temperature rising trend, and each wave trough point represents a bottom point of a temperature falling trend, and the wave peak sequence and the wave trough sequence are processed respectively, the time interval between each two adjacent wave peak points is calculated to obtain a wave peak interval value, and the time interval between each two adjacent wave trough points is calculated to obtain a wave trough interval value.

4. The audio-video stream real-time face recognition intelligent tracking and labeling system according to claim 3, characterized in that, Based on the calculated interval values, it is determined whether the interval values fall within a preset typical time range conforming to human respiratory rhythm, and whether the difference between the longest interval value and the shortest interval value is less than a maximum allowed fluctuation range threshold. If most of the interval values are within the typical time range and the maximum difference between the longest interval value and the shortest interval value does not exceed the maximum allowed fluctuation range threshold, it is determined that the fluctuations have regularity. If the regularity condition is met, it is determined that the temperature fluctuations are periodic fluctuations related to respiratory activity, the average of all interval values is calculated as the average fluctuation period, the average amplitude of the peaks and valleys is recorded as the fluctuation strength, the average fluctuation period and the fluctuation strength are taken as the rhythm characteristics of the periodic fluctuations, and the rhythm characteristics are taken as the respiratory thermal disturbance signal reflecting the target respiratory rhythm.

5. The audio-video stream real-time face recognition intelligent tracking and labeling system according to claim 2, characterized in that, Based on the target region, the boundary coordinates of the target region in the thermal imaging data are obtained, and the spatial coordinate mapping relationship is used to convert the target region center point pixel coordinates in the thermal imaging data stream to a unit observation direction vector in the thermal imaging sensor coordinate system based on the pre-stored spatial calibration parameters of the infrared thermal imaging sensor and the infrared spectrum sensor. The unit observation direction vector is converted to the infrared spectrum sensor coordinate system through coordinate conversion, and the direction vector is converted to the instruction parameters required by the infrared spectrum sensor according to the type of the infrared spectrum sensor to determine the target corresponding detection region in the infrared spectrum data for analysis. According to the characteristic absorption wavelength of the specific component to be detected in the target exhaled gas, a plurality of wave bands contained in the infrared spectrum data stream are selected as core observation wave bands for analyzing the gas concentration change. In the continuously collected infrared spectrum data stream, the signal intensity values of the selected characteristic absorption wavelength adjacent wave bands are extracted in the target corresponding detection region in real time, and the signal intensity values are sequentially recorded according to the time stamps synchronized with the thermal imaging data to form a characteristic wave band intensity original sequence. The characteristic wave band intensity original sequence is preprocessed, and the preprocessing includes baseline correction, trend removal and noise suppression operation. After preprocessing the characteristic wave band intensity original sequence, a corrected intensity sequence is obtained.

6. The audio-video stream real-time face recognition intelligent tracking and labeling system according to claim 5, characterized in that, The corrected intensity sequence is analyzed to detect the periodic fluctuation component, and the periodic fluctuation component is matched and verified with the period of the respiratory thermal disturbance signal analyzed from the thermal imaging data stream. When the two periods match, the periodic intensity fluctuation component that has passed the period matching verification is extracted as a time-series spectrum fluctuation signal reflecting the gas exchange dynamics. The two kinds of dynamic physiological signals, the respiratory thermal disturbance signal and the time-series spectrum fluctuation signal, are output. 7.The audio-video stream real-time face recognition and intelligent tracking and labeling system of claim 1, wherein, The feature fusion module extracts features from the two kinds of dynamic physiological signals. For the respiratory thermal disturbance signal analyzed from the thermal imaging data stream, the time sequence features reflecting the respiratory rhythm are extracted, including periodic fluctuations, period values, amplitude values and stability parameters. Meanwhile, the spatial heat field distribution characteristics corresponding to the respiratory heat disturbance signal are extracted, including core spatial position positioning and spatial diffusion range determination; For the time-series spectral fluctuation signal parsed from the spectral data stream, its time-series characteristics reflecting gas exchange dynamics are extracted, including periodic fluctuation identification, fluctuation period, and wave peak-to-valley intensity ratio; Meanwhile, its corresponding spatial characteristics are extracted, including determining the detection region position identifier and converting it to standard spatial coordinates, determining the detection region position identifier by directly reading the unique identifier of the target corresponding detection region in the spectral sensor when generating the time-series spectral fluctuation signal, and converting the unique identifier to a set of reference coordinates in the thermal imaging field of view coordinate system; Meanwhile, its corresponding spatial characteristics are extracted, including determining the detection region position identifier and converting it to standard spatial coordinates, determining the detection region position identifier by directly reading the unique identifier of the target corresponding detection region in the spectral sensor when generating the time-series spectral fluctuation signal, and converting the unique identifier to a set of reference coordinates in the thermal imaging field of view coordinate system; After completing the feature extraction, the time-series characteristic sequence of the respiratory heat disturbance signal and the time-series characteristic sequence of the spectral fluctuation signal are aligned on the time axis to ensure that the data points representing the same physiological event in the two signal sequences are completely matched in time, forming a time-aligned time-series rhythm characteristic combination. 8.The audio-video stream real-time face recognition and intelligent tracking and labeling system of claim 7, wherein, The time-aligned time-series rhythm characteristic combination is spliced with the spatial distribution characteristics extracted from the homologous signal to connect the time-series rhythm characteristics of the respiratory heat disturbance signal with its spatial heat field distribution characteristics at the beginning and end, and to connect the time-series rhythm characteristics of the spectral fluctuation signal with its spatial position characteristics at the beginning and end. The two groups of connected characteristics are spliced again to form a long feature vector; After splicing, the long feature vector formed by splicing is weighted, and the feature values in different dimensions of the long feature vector are scaled based on the preset weight coefficients, which are set based on the contribution of each feature to distinguishing different individuals in historical data. The weighted composite feature vector is used as the physiological feature identifier of the target. 9.The audio-video stream real-time face recognition and intelligent tracking and labeling system of claim 1, wherein, The identity recognition module assigns a unique identity identifier to the target when the physiological feature identifiers generated in the continuous time period show a high degree of consistency during continuous monitoring of the target area; The identity recognition module assigns a unique identity identifier to the target when the physiological feature identifiers generated in the continuous time period show a high degree of consistency during continuous monitoring of the target area; 10.The audio-video stream real-time face recognition and intelligent tracking and labeling system of claim 1, wherein, The identity recognition module assigns a unique identity identifier to the target when the physiological feature identifiers generated in the continuous time period show a high degree of consistency during continuous monitoring of the target area; The identity recognition module assigns a unique identity identifier to the target when the physiological feature identifiers generated in the continuous time period show a high degree of consistency during continuous monitoring of the target area; The visible light video stream frame is aligned with the time point of the identity identifier and the spatial position sequence, the target real-time position of the spatial position coordinates and the identity identifier in the continuous trajectory data is mapped into the corresponding pixel coordinates of the current visible light video frame, a labeling graphic element is generated based on the pixel coordinates obtained by the mapping, including an identity identifier and a trajectory line connecting historical position points, the labeling graphic element is rendered to the corresponding coordinate position in real time through an image superposition technology, a labeling picture is formed, and the trajectory line is continuously updated in the subsequent frames of the video stream to draw the latest moving path of the target.

Images