Automatic camera tracking method and device based on machine vision

Abstract

The invention relates to the technical field of machine vision and intelligent monitoring, and discloses an automatic camera tracking method and device based on machine vision, and the method comprises the steps: obtaining a real-time video frame sequence, and carrying out the preprocessing of the real-time video frame sequence, and obtaining an initial image set; identifying features of the initial image set, judging an abnormal state, and fusing the data to generate behavior change trend data; a smooth trajectory is predicted based on the data, and a parameter adjustment data set is obtained by combining camera position calculation deviation; adjusting camera parameters according to the generation instruction, detecting shielding and fusing trajectory data to obtain real-time tracking parameters; and applying parameters, complementing missing frame verification, updating a process and reextracting features, and performing time sequence analysis in case of abnormality to obtain a final tracking logic sequence. According to the method, accurate target locking can be realized, and the stability and accuracy of abnormal emotion tracking in a crowded environment are improved.

Description

Technical Field

[0001] This invention relates to the fields of machine vision and intelligent monitoring technology, and in particular to a machine vision-based automatic camera tracking method and apparatus. Background Technology

[0002] Currently, in the fields of intelligent surveillance and human-computer interaction, machine vision-based automatic tracking technology is crucial due to its widespread application in security monitoring, motion analysis, and smart devices. This technology captures and tracks the movement of targets in real time using cameras, enabling precise positioning in dynamic scenarios. Especially in crowded public places such as shopping malls and train stations, it can assist in achieving core needs such as security control and service optimization, representing an important development direction in the current field of visual inspection applications.

[0003] In existing technologies, automatic tracking methods typically rely on the static features of the target (such as appearance and outline) or a single motion trajectory for tracking. For example, some systems identify the target by recognizing the color of the clothing and the outline of the body, and then predict the movement path based on simple displacement calculations; other technologies obtain the target's motion direction by comparing pixels between consecutive frames, and then adjust camera parameters accordingly. However, these methods have significant limitations in complex dynamic scenes: facing background noise interference and frequent occlusion between targets in crowded environments, existing technologies struggle to effectively separate the target from the background information, easily leading to tracking interruptions; and they ignore the impact of the target's emotional state on behavior. When the target makes sudden movements due to emotional changes (such as tension or panic), the system cannot adjust the tracking strategy in time, resulting in trajectory prediction errors, tracking lag, or even missing the target.

[0004] In existing technologies, the lack of a dynamic correlation mechanism between "emotion-behavior-tracking parameters," coupled with a simplistic occlusion handling strategy and insufficient flexibility in adjusting camera parameters, makes it difficult for automatic tracking technologies to meet the stability and accuracy requirements in crowded environments. Therefore, existing technologies struggle to achieve accurate target locking and continuous tracking. Summary of the Invention

[0005] This invention provides a machine vision-based automatic camera tracking method and apparatus to solve the problem that existing technologies struggle to achieve accurate target locking and continuous tracking.

[0006] In a first aspect, to solve the above-mentioned technical problems, the present invention provides a machine vision-based automatic camera tracking method, comprising: Acquire real-time video frame sequences, perform image preprocessing, and obtain an initial image set containing the features of the target object; Feature recognition is performed on the initial image set to extract the facial expression and posture features of the target object and analyze its state. If the state is determined to be a preset abnormal state, data fusion is performed on the initial image set to generate behavioral change trend data. Based on the behavioral change trend data, trajectory prediction and smoothing are performed to determine the trajectory data. The trajectory deviation is calculated by combining the current position parameters of the camera to obtain the camera parameter adjustment dataset. Based on the camera parameters, the dataset is adjusted to generate control commands, which adjust the camera's operating parameters. At the same time, occlusion detection is performed on the real-time video frame sequence to obtain occlusion data. The trajectory data is then fused to obtain real-time tracking parameters. The real-time tracking parameters are applied to the processing loop of the real-time video frame sequence. If frame data loss due to occlusion is detected, linear interpolation is used to fill the missing frame data and data consistency is checked to obtain continuous tracking data. The processing flow based on the continuous tracking data updates the real-time video frame sequence, re-extracts the target object features and updates the state. If it is still a preset abnormal state, new behavior change trend data is generated and time series analysis is performed to obtain the final tracking logic sequence.

[0007] In one optional implementation, the step of acquiring a real-time video frame sequence, performing image preprocessing, and obtaining an initial image set containing the features of the target object includes: Acquire the real-time video frame sequence captured by the camera; Image data of the target person is obtained from the real-time video frame sequence, and first image data containing the features of the target object is separated by face detection and human body contour segmentation. For the first image data, background noise is removed by median filtering to generate the second image data; For the second image data, histogram equalization is used to enhance image contrast, generating third image data; For the third image data, facial expression features and body posture features are extracted through key point detection to generate initial facial images and initial body images, forming an initial image set.

[0008] In one optional implementation, the initial image set is subjected to feature recognition to extract facial expression and posture features of the target object and analyze its state. If the state is determined to be a preset abnormal state, the initial image set is then fused to generate behavioral change trend data, including: For the initial facial images in the initial image set, a convolutional neural network is used to extract facial expression features, analyze the degree of eyebrow raising and the frequency of eye squinting, and generate expression feature data. For the facial expression feature data, if the eyebrow raising amplitude is greater than a preset amplitude threshold or the eye squinting frequency exceeds a preset frequency threshold, it is judged as a preset abnormal state and an abnormal emotion label is generated. Based on the abnormal emotion label, the human posture key point sequence is extracted from the initial limb images in the initial image set through key point detection, the displacement vector between key points is calculated, and displacement vector data is generated. The displacement vector data is combined with the facial expression feature data and weighted fusion is used to generate behavioral change trend data.

[0009] In one optional implementation, the step of performing trajectory prediction and smoothing processing based on the behavioral change trend data to determine trajectory data, and calculating the trajectory deviation in conjunction with the current position parameters of the camera to obtain a camera parameter adjustment dataset includes: Obtain the current position parameters of the camera; The dynamic features of the behavioral change trend data are extracted using a convolutional neural network, and the abnormal emotion labels are combined to predict the trajectory of the target object. The movement trajectory is smoothed using a Bézier curve algorithm to determine the trajectory data; The deviation between the trajectory data and the current position parameters of the camera is calculated. If the deviation exceeds a preset deviation threshold, the trajectory deviation is obtained by calculating the Euclidean distance, and correction data is generated. Based on the correction data, linear interpolation is used to calculate the adjustment values ​​of the camera translation and scaling parameters, generating a camera parameter adjustment dataset.

[0010] In one optional implementation, the step of generating control commands based on the camera parameter adjustment dataset, adjusting the camera operating parameters, performing occlusion detection on the real-time video frame sequence after parameter adjustment to obtain occlusion data, and fusing the trajectory data to obtain real-time tracking parameters includes: Based on the camera parameters, adjust the dataset to generate control commands that conform to the camera control protocol, and drive the camera to adjust the shooting angle and focal length. For the real-time video frame sequence captured by the adjusted camera, the occlusion area features are extracted by pixel density comparison to generate occlusion detection data; If the occlusion detection data exceeds a preset occlusion threshold, the trajectory data and the occlusion detection data are fused using a linear interpolation method to generate a first tracking parameter; Based on the first tracking parameter and the abnormal emotion label, the weight of the first tracking parameter is adjusted by weighted average to obtain the real-time tracking parameter.

[0011] In one optional implementation, the step of applying the real-time tracking parameters to the processing loop of the real-time video frame sequence, if frame data loss due to occlusion is detected, then linear interpolation is used to fill the missing frame data and data consistency verification is performed to obtain continuous tracking data, including: The real-time tracking parameters are applied to the processing loop of the real-time video frame sequence, the shooting angle is adjusted by the camera control unit, and an optimized video frame sequence is generated; Perform frame integrity detection on the optimized video frame sequence to determine whether there are missing frames caused by occlusion; If the missing frame exists, based on the trajectory data, determine the position coordinates and morphological features of the target object in the valid frames before and after the missing frame, use linear interpolation to calculate the key feature parameters of the target object in the missing frame, fill the missing frame data, and obtain the filled video frame sequence. The data consistency of the padded video frame sequence is checked, and the positional deviation and morphological feature variation of the target object between adjacent frames are calculated. If the change in position deviation and morphological features exceeds a preset continuity threshold, the key feature parameters are re-optimized based on the trajectory data until the preset continuity threshold is met, thereby generating continuous tracking data.

[0012] In one optional implementation, the process of updating the real-time video frame sequence based on the continuous tracking data, re-extracting target object features and updating the state, and if it is still a preset abnormal state, generating new behavioral change trend data and performing time-series analysis to obtain the final tracking logic sequence, includes: The processing flow based on the continuous tracking data updates the real-time video frame sequence, and the updated facial expression features and posture features are re-extracted from the real-time video frame sequence; Based on the updated facial expression and posture features, it is determined whether the updated state of the target object belongs to a preset abnormal state. If the updated state is still a preset abnormal state, new behavioral change trend data is generated. The new behavioral change trend data is analyzed over time to generate abnormal emotion tracking logic, resulting in the final tracking logic sequence.

[0013] Secondly, the present invention provides a camera-based automatic tracking device based on machine vision, comprising: The data acquisition module acquires real-time video frame sequences, performs image preprocessing, and obtains an initial image set containing the features of the target object. The feature analysis module performs feature recognition on the initial image set, extracts the facial expression features and posture features of the target object and analyzes the state. If the state is determined to be a preset abnormal state, the initial image set is fused to generate behavioral change trend data. The trajectory prediction module performs trajectory prediction and smoothing processing based on the behavior change trend data, determines the trajectory data, calculates the trajectory deviation by combining the current position parameters of the camera, and obtains the camera parameter adjustment dataset. The control and tracking module generates control commands based on the camera parameter adjustment dataset, adjusts the camera operating parameters, and simultaneously performs occlusion detection on the real-time video frame sequence to obtain occlusion data. It then fuses the trajectory data to obtain real-time tracking parameters. The frame loss compensation module applies the real-time tracking parameters to the processing loop of the real-time video frame sequence. If frame data loss due to occlusion is detected, linear interpolation is used to fill the missing frame data and data consistency verification is performed to obtain continuous tracking data. The logic output module updates the real-time video frame sequence based on the continuous tracking data, re-extracts the target object features and updates the state. If it is still a preset abnormal state, it generates new behavior change trend data and performs time series analysis to obtain the final tracking logic sequence.

[0014] Thirdly, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the machine vision-based automatic camera tracking method described in any one of the preceding claims.

[0015] Fourthly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the automatic camera tracking method based on machine vision described in any one of the above-described methods.

[0016] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention uses a preprocessing flow of “video frame acquisition - target separation - noise removal - contrast enhancement - key point extraction”, combined with face detection, median filtering and other technologies, to filter redundant interference and provide high-quality expression and posture feature data for subsequent analysis, thus solving the problem that target feature extraction in the prior art is easily affected by the background.

[0017] (2) This invention combines facial expression recognition with body posture analysis. It extracts facial expression features through convolutional neural networks to judge abnormal emotions, integrates the displacement vectors of key limb points to generate behavioral trend data, and uses emotions as trajectory prediction variables to reduce the prediction bias of sudden behaviors caused by emotions and improve the accuracy of behavior prediction.

[0018] (3) The present invention uses pixel density comparison to identify occluded areas, and uses linear interpolation to fill in the missing frames of occlusion in combination with trajectory data. The continuity is ensured by data consistency verification, avoiding the tracking interruption caused by occlusion in the prior art, and ensuring continuous target locking.

[0019] (4) The present invention calculates the camera parameter adjustment value based on behavioral trend data, integrates occlusion detection to optimize tracking parameters, and forms a dynamic adjustment mechanism. The parameters can be adjusted in real time according to the target behavior and environmental occlusion, solving the problem of fixed parameters and poor adaptability in the existing technology, and ensuring stable tracking in crowded environments. Attached Figure Description

[0020] Figure 1 This is a schematic flowchart of the automatic camera tracking method based on machine vision provided in the first embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the automatic camera tracking device based on machine vision provided in the second embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Reference Figure 1 The first embodiment of the present invention provides a machine vision-based automatic camera tracking method, comprising the following steps: S11, acquire the real-time video frame sequence, perform image preprocessing, and obtain an initial image set containing the features of the target object; S12, perform feature recognition on the initial image set, extract the facial expression features and posture features of the target object and analyze the state. If the state is determined to be a preset abnormal state, perform data fusion on the initial image set to generate behavioral change trend data. S13, Based on the behavior change trend data, perform trajectory prediction and smoothing processing to determine the trajectory data, and calculate the trajectory deviation by combining the current position parameters of the camera to obtain the camera parameter adjustment dataset; S14, generate control commands based on the camera parameter adjustment dataset, adjust camera operating parameters, and simultaneously perform occlusion detection on the real-time video frame sequence to obtain occlusion data, and fuse the trajectory data to obtain real-time tracking parameters; S15, the real-time tracking parameters are applied to the processing loop of the real-time video frame sequence. If frame data loss due to occlusion is detected, linear interpolation is used to fill the missing frame data and data consistency is checked to obtain continuous tracking data. S16, based on the continuous tracking data, update the real-time video frame sequence processing flow, re-extract the target object features and update the state. If it is still a preset abnormal state, generate new behavior change trend data and perform time series analysis to obtain the final tracking logic sequence.

[0023] In step S11, it is necessary to acquire a real-time video frame sequence, perform image preprocessing, and obtain an initial image set containing the features of the target object, including: Acquire the real-time video frame sequence captured by the camera; Image data of the target person is obtained from the real-time video frame sequence, and first image data containing the features of the target object is separated by face detection and human body contour segmentation. For the first image data, background noise is removed by median filtering to generate the second image data; For the second image data, histogram equalization is used to enhance image contrast, generating third image data; For the third image data, facial expression features and body posture features are extracted through key point detection to generate initial facial images and initial body images, forming an initial image set.

[0024] It should be noted that the target's characteristics include two core types of information: facial expression features and body posture features. Facial expression features specifically refer to facial details that reflect emotional fluctuations, such as the degree of eyebrow raising and the frequency of eye squinting. For example, a significant raising of the eyebrows may correspond to surprise, while frequent eye squinting may suggest tension. Body posture features cover the position coordinates, movement angles, and trajectories of key parts of the body, such as the shoulders, elbows, and wrists. For example, rapid arm swings and sudden body turns can help determine the target's behavioral intentions.

[0025] In this step, the real-time video stream is first received through the camera's hardware interface (such as a network interface or USB interface). The video stream is then broken down into continuous single-frame images according to a preset frame rate (common industry standards such as 25fps for PAL and 30fps for NTSC) to form a real-time video frame sequence. At the same time, the acquisition timestamp of each frame is recorded to ensure the timing accuracy of subsequent processing.

[0026] Furthermore, face detection and human contour segmentation are used to separate target object data. Face detection first extracts multi-scale features from a single frame image using the deep learning-based MTCNN algorithm to generate candidate facial regions. Then, a classifier filters out real faces, and finally, bounding box regression outputs accurate facial bounding box coordinates (e.g., top left corner). bottom right corner Human body contour segmentation uses the Mask R-CNN algorithm to perform pixel-level classification of images, marking the pixels of the target object's limbs as foreground and other pixels as background. The limb contour is delineated based on the connectivity of the foreground pixels. Finally, the facial bounding box and the limb contour are combined to crop out the first image data containing only the target object's face and limbs.

[0027] In one implementation, median filtering is also required to denoise the first image data. A 3×3 or 5×5 filtering window is used (selected according to the image resolution; a 3×3 window is commonly used for 1920x1080 resolution). For each pixel, the median of the gray values ​​of all pixels in the window is taken to replace the original pixel value. If it is a color image, the RGB three channels are processed separately to effectively remove background noise such as light flickering and crowd movement, and generate the second image data.

[0028] Furthermore, histogram equalization is performed on the second image data. First, the gray-level histogram of the image is statistically analyzed, the cumulative distribution function of gray level is calculated and normalized to the range of 0-255, and then a gray-level mapping relationship is established based on the cumulative distribution function to redistribute the pixel gray-level values. If it is a color image, it is first converted to YUV space, and only the luminance channel (Y channel) is processed before being converted back to RGB to solve the problem of low contrast caused by uneven lighting and generate the third image data.

[0029] In addition, by using keypoint detection algorithms such as OpenPose to process the third image data, 17 key points on the face (such as the eyebrow tip, the corner of the eye, and the corner of the mouth) and 15 key points on the limbs (such as the shoulder, elbow, and wrist) are located. The coordinate information of these key points is extracted, and initial facial images and initial limb images containing facial key points and limb key points are generated respectively, which are then combined into an initial image set.

[0030] It's worth noting that the choice of algorithm for face detection needs to balance accuracy and speed. The MTCNN algorithm, with its three-layer network structure of P-Net, R-Net, and O-Net, can meet real-time requirements while ensuring detection accuracy. For human contour segmentation, the Mask R-CNN algorithm is used, which can achieve pixel-level segmentation, ensuring the accuracy of limb contours. The median filter window size should not be too large, otherwise it will blur image edge details; a 3×3 window provides the best noise removal effect for most surveillance scenarios. When processing color images with histogram equalization, directly processing the RGB channels can easily lead to color distortion. Converting to YUV space and processing the luminance channel separately can enhance contrast while preserving the original color.

[0031] For example, taking a shopping mall surveillance scenario, the camera captures a real-time video frame sequence with a resolution of 1920x1080 and a frame rate of 25fps. The MTCNN algorithm is used to process a single frame image: P-Net generates multiple candidate facial regions, R-Net filters and refines the bounding boxes, and O-Net outputs the coordinates of the target customer's facial bounding box. =450, =280, =650, =480); At the same time, the Mask R-CNN algorithm was used to segment the frame at the pixel level to outline the customer's limb contour from head to waist, and after cropping, the first image data containing the face and limbs was obtained.

[0032] Furthermore, due to the salt-and-pepper noise generated by the flickering lights in the shopping mall, a 3×3 window median filter was applied to the first image data. Taking the pixel at coordinates (550, 380) as an example, the grayscale values ​​within the window were 118, 120, 122, 121, 123, 125, 124, 126, and 127. After sorting, the median value of 123 was used to replace the original pixel value, resulting in second image data with no obvious noise. Because the lighting in some parts of the shopping mall was relatively dim, the grayscale values ​​of the second image data were concentrated between 90 and 140. Through histogram equalization, the grayscale value 90 was mapped to 0, 140 to 255, and the intermediate grayscale values ​​were mapped proportionally. After processing, the grayscale range was expanded to 0-255, and details such as facial wrinkles and limb joints became clearer, resulting in the third image data. Finally, the OpenPose algorithm was used to extract key points from the third image data. Facial key points included 17 points such as the left eyebrow endpoint (480,310) and the right corner of the eye (620,330). Limb key points included 15 points such as the left shoulder (460,390) and the right elbow (670,460). Initial facial images and initial limb images were generated respectively to form the initial image set.

[0033] In step S12, feature recognition needs to be performed on the initial image set to extract the facial expression and posture features of the target object and analyze its state. If the state is determined to be a preset abnormal state, data fusion is performed on the initial image set to generate behavioral change trend data, including: For the initial facial images in the initial image set, a convolutional neural network is used to extract facial expression features, analyze the degree of eyebrow raising and the frequency of eye squinting, and generate expression feature data. For the facial expression feature data, if the eyebrow raising amplitude is greater than a preset amplitude threshold or the eye squinting frequency exceeds a preset frequency threshold, it is judged as a preset abnormal state and an abnormal emotion label is generated. Based on the abnormal emotion label, the human posture key point sequence is extracted from the initial limb images in the initial image set through key point detection, the displacement vector between key points is calculated, and displacement vector data is generated. The displacement vector data is combined with the facial expression feature data and weighted fusion is used to generate behavioral change trend data.

[0034] It should be noted that convolutional neural networks (such as VGG-Face) are the core tools for extracting facial expression features. Through a structure of 13 convolutional layers, 5 pooling layers and 3 fully connected layers, they extract features layer by layer from the initial facial image, gradually abstracting from low-level edge and texture features into high-level feature vectors that can represent the degree of eyebrow raising and the frequency of eye squinting, i.e., expression feature data.

[0035] The preset amplitude and frequency thresholds are determined by technicians or experts in the field of facial expression research through a large number of sample statistics. For example, in common monitoring scenarios, the threshold for eyebrow raising amplitude is set to 4 mm, and the threshold for eye squinting frequency is set to 2 times per second. Exceeding either threshold is judged as a preset abnormal state. These thresholds can be dynamically adjusted by those skilled in the art according to the monitoring and tracking needs in different scenarios. The values ​​involved in this article are only examples.

[0036] Furthermore, those skilled in the art will understand that the above-mentioned criteria for judging abnormal states are merely illustrative. In practical applications, more feature dimensions (such as mouth opening degree, head posture, and whole-body motion energy) can be added, and more complex classification models (such as SVM and LSTM) can be used for comprehensive judgment. When the main features (such as facial features) fail due to occlusion or lighting conditions, the system can automatically switch to relying on secondary features (such as limb posture) or adopt a prediction mode based on motion history to maintain the continuity of tracking. This mechanism based on multimodal information fusion and fault switching is a conventional extension and application based on the core idea of ​​this invention.

[0037] It's worth noting that obtaining the eyebrow raising amplitude also involves the conversion from pixel displacement to physical displacement. In one possible implementation, it's necessary to first obtain camera parameters (such as a 1 / 2.3-inch sensor commonly used in surveillance scenarios, a focal length of 6.3-18.9mm, etc., or obtain the intrinsic parameter matrix through calibration), and determine the actual distance between the target person and the camera (e.g., obtained through a camera range sensor, often 3-8 meters in shopping mall scenarios); then, using algorithms such as OpenPose and AlphaPose that can achieve key point detection, locate the pixel coordinates of the key points of the eyebrows before and after the raising, and calculate the vertical pixel difference; then, combining the camera intrinsic parameters and the actual distance, convert the pixel difference into physical displacement using a formula (e.g., the pixel difference divided by the quotient of the product of the focal length and the pixel density, then multiplied by the actual distance).

[0038] Human posture keypoint sequence is a set of limb keypoint coordinates arranged in chronological order across consecutive frames, such as a set of keypoint coordinates per second of 10 frames. Displacement vectors, obtained by calculating the coordinate differences of the same keypoint in adjacent frames, reflect the direction and amplitude of limb movements. During weighted fusion, to highlight the influence of emotion on behavior, the weight of facial expression feature data can be set to 0.6, and the weight of displacement vector data can be set to 0.4. A comprehensive behavioral change trend data is obtained through weighted summation. These weight values ​​can be dynamically adjusted by those skilled in the art according to the monitoring and tracking needs of different scenarios; the values ​​mentioned in this article are merely examples.

[0039] In this step, the initial facial image is first scaled to the size required by the convolutional neural network (such as the VGG-Face model) (e.g., 224×224), and the pixel values ​​are normalized (mapped to the 0-1 range) before being input into the model. The model captures facial details through multiple convolution and pooling operations, and the fully connected layer outputs a 4096-dimensional feature vector. From this vector, eyebrow and eye-related feature components are extracted. Combined with the coordinates of the key points of the eyebrows and eyes in the initial facial image, the eyebrow raising amplitude (e.g., the vertical distance between the left eyebrow endpoint and the initial position) and the eye squinting frequency (the number of times the distance between the key points of the upper and lower eyelids is less than 10 pixels per unit time) are calculated to generate expression feature data (e.g., eyebrow raising amplitude of 5 mm, eye squinting frequency of 3 times / second).

[0040] Furthermore, the facial expression feature data is compared with preset amplitude and frequency thresholds. If the eyebrow raising amplitude is greater than 4 mm or the eye squinting frequency is greater than 2 times / second, it is judged as a preset abnormal state. Abnormal emotion labels are generated according to the degree of exceeding the amplitude or frequency thresholds, such as "Tense - Moderate" and "Anxious - Severe". Next, based on the abnormal emotion labels, keypoint detection (such as the AlphaPose algorithm) is performed on the initial limb image to extract a sequence of human posture keypoints for 10 consecutive frames (each frame contains 15 keypoint coordinates). The displacement vector of the same keypoint in adjacent frames is calculated, such as... Right wrist coordinates at time (500, 450) The time coordinates are (520, 470), and the displacement vector is (20, 20) pixels. Displacement vector data containing the displacement vectors of all key points is generated. Finally, the facial expression feature data and displacement vector data are quantized (mapped to the range of 0-10), and weighted and summed with weights of 0.6 and 0.4. For example, if the quantized facial expression feature data is [5, 3] and the displacement vector data is [6, 4], then the behavior change trend data is [5×0.6+6×0.4,3×0.6+4×0.4]=[5.4,3.4].

[0041] It is worth noting that the VGG-Face model is highly robust to changes in lighting, making it suitable for facial expression extraction in complex surveillance scenarios. The AlphaPose algorithm can accurately locate limb keypoints even with slight occlusion, ensuring the accuracy of displacement vector calculations. Furthermore, the models and algorithms mentioned in this paper are merely examples of possible implementations for understanding by those not skilled in the art; all can be based on publicly available pre-trained models. Those skilled in the art will know that public datasets (such as FER2013 for facial expressions and COCO for poses) or self-built datasets can be used to fine-tune the final fully connected layer of the pre-trained model according to the specific needs of the actual application scenario, achieving the effects of this invention. This process is a standard practice in the field of deep learning. The weights of the weighted fusion can be adjusted according to the scenario. For example, in security scenarios where high sensitivity to emotions is required, the weight of facial expression feature data can be increased to 0.7; in motion analysis scenarios, the weight of displacement vector data can be increased to 0.5.

[0042] For example, in a shopping mall surveillance scenario, the initial facial image (224×224 resolution) is processed using the VGG-Face model. The model combines camera intrinsic parameters (such as focal length, distance between the target and the camera) and the pixel coordinates of key facial points to convert the pixel difference of eyebrow raising into a physical amplitude of 5 mm. At the same time, the frequency of eye squinting is detected as 3 times per second, generating facial expression feature data [5,3] (units are mm and times / second, respectively). This data is compared with preset amplitude thresholds and frequency thresholds (4 mm and 2 times / second) to determine a preset abnormal state and generate an abnormal emotion label of "tense-moderate". Based on the label, the AlphaPose algorithm is used to extract the key point sequence of 10 consecutive frames of the initial limb image, and the displacement vector of the right shoulder (15,10) pixels, the displacement vector of the right elbow (25,15) pixels, etc. are calculated. First, the average pixel value of the displacement vector of the key parts is taken (e.g., (15+25) / 2=20 pixels, (10+15) / 2=12.5 pixels). Then, combined with the same camera parameters and distance (the camera intrinsic parameters can be obtained through conventional methods such as Zhang Zhengyou calibration, and the distance can be obtained through monocular ranging or sensors), the pixel displacement is converted into millimeter-level physical displacement (e.g., 20 pixels correspond to 20 millimeters, 12.5 pixels correspond to 12.5 millimeters) by multiplying the physical displacement by the product of the pixel difference and the focal length and pixel density, and then multiplying by the actual distance calculation method. The displacement is generated as [20,12.5] (in millimeters). The pixel density is obtained through camera intrinsic parameter calibration. Subsequently, the facial expression feature data [5,3] and displacement vector data [20,12.5] were subjected to Min-Max standardization (mapped to a range of 0-10, such as 5 for eyebrow raising amplitude of 5 mm, 3 for eye squinting frequency of 3 times / second, 8 for limb displacement of 20 mm, and 5 for limb displacement of 12.5 mm), resulting in standardized data [5,3] and [8,5]. Finally, the data were weighted and fused with a weight of 0.6 (facial expression feature) and 0.4 (limb displacement) to obtain behavioral change trend data [5×0.6+8×0.4,3×0.6+5×0.4]=[5.2,3.8], reflecting the behavioral trend of customers with large limb movements accompanied by tension.

[0043] In step S13, trajectory prediction and smoothing processing are performed based on the behavior change trend data to determine the trajectory data. The trajectory deviation is calculated by combining the current camera position parameters to obtain the camera parameter adjustment dataset, including: Obtain the current position parameters of the camera; The dynamic features of the behavioral change trend data are extracted using a convolutional neural network, and the abnormal emotion labels are combined to predict the trajectory of the target object. The movement trajectory is smoothed using a Bézier curve algorithm to determine the trajectory data; The deviation between the trajectory data and the current position parameters of the camera is calculated. If the deviation exceeds a preset deviation threshold, the trajectory deviation is obtained by calculating the Euclidean distance, and correction data is generated. Based on the correction data, linear interpolation is used to calculate the adjustment values ​​of the camera translation and scaling parameters, generating a camera parameter adjustment dataset.

[0044] It's important to note that the camera's current position parameters include physical coordinates (such as X, Y, and Z axis coordinates in the monitored scene), shooting angle (horizontal and vertical angles), and focal length. These parameters are acquired through the camera's built-in sensors and serve as the benchmark for calculating trajectory deviation. Convolutional neural networks (such as temporal convolutional networks, TCNs) extract dynamic features from behavioral change trend data, such as gait frequency and limb swing amplitude. Combined with abnormal emotional labels (such as "nervousness" corresponding to rapid movement), they predict the target object's movement trajectory for the next 5-10 seconds through temporal analysis. The Bézier curve algorithm adjusts the trajectory curvature by setting 2-3 control points, eliminating jump points in the trajectory and making it smoother and more continuous.

[0045] The preset deviation threshold is set based on the average value of historical test data from similar scenarios. Its function is to determine whether the deviation between the predicted target trajectory and the current camera position is too large, thereby triggering trajectory correction to ensure tracking accuracy. The threshold is set according to the monitoring range, commonly 2 meters, but can be dynamically adjusted by those skilled in the art based on the monitoring and tracking needs of different scenarios. The values ​​mentioned in this article are merely examples. Euclidean distance is used to calculate the straight-line distance between the target position and the center of the camera's field of view in the trajectory data, i.e., the trajectory deviation, and based on this, correction data is generated (e.g., adjusting 2 meters to the left, 1 meter downwards). Linear interpolation, based on the correction data, inserts intermediate values ​​between the current parameters and the target parameters to calculate the step-by-step adjustment values ​​for camera translation (horizontal and vertical movement distance) and zoom (focal length adjustment value), forming a parameter adjustment dataset.

[0046] In this step, the current position parameters are first read through the camera's control interface, such as physical coordinates (8,5,3) meters, horizontal angle 0°, vertical angle 0°, and focal length 30 mm. Then, the behavioral change trend data is input into the TCN network. The network extracts time series features through multi-layer convolution, such as extracting a gait frequency of 2 steps / second and an arm swing amplitude of 30 cm from the data [5.2,3.8]. Combined with the abnormal emotion label of "tense-moderate", it predicts that the target will move to the left in the next 5 seconds at a speed of 1.2 m / s, generating a movement trajectory containing 5 key points (A(8,5,3), B(9.2,5,3), C(10.4,5,3), D(11.6,5,3), E(12.8,5,3)). Next, the trajectory was smoothed using the Bézier curve algorithm. Control points (8.6,5,3) and (11.2,5,3) were set. After adjustment, the key points of the trajectory became A(8,5,3), B'(9.1,5,3), C'(10.5,5,3), D'(11.7,5,3), and E(12.8,5,3), eliminating the small fluctuations in the original trajectory and determining the trajectory data.

[0047] Furthermore, the deviation between the trajectory data and the current position parameters of the camera is calculated. The physical coordinates corresponding to the center of the camera's field of view are (8, 5, 3). The Euclidean distance between the midpoint C' (10.5, 5, 3) of the trajectory and the center of the field of view is... For each meter, exceeding a preset deviation threshold of 2 meters, correction data is generated: "Adjust horizontally to the left by 2.5 meters". Finally, linear interpolation is used to divide the 2.5-meter translation distance into two steps (each step is 1.25 meters). The focal length is adjusted from 30 mm to 35 mm according to the change in target distance, generating a camera parameter adjustment dataset. For example, the first step (0-2 seconds): move horizontally to the left by 1.25 meters, focal length 32 mm; the second step (2-4 seconds): move horizontally to the left by 1.25 meters, focal length 35 mm.

[0048] It should be noted that the current position parameters of the camera typically include spatial coordinates, angle, and focal length; when calculating the deviation between the trajectory data and the current position parameters of the camera, only the spatial coordinates of the current position parameters of the camera, that is, the physical coordinates corresponding to the center of the camera's field of view, are used, rather than all position parameters.

[0049] For example, in a shopping mall surveillance scenario, the camera's current position parameters are physical coordinates (10,6,4) meters, horizontal angle 0°, vertical angle 0°, and focal length 28 mm. Behavioral change trend data is [7.6,8.2]. Combined with the "anxiety-severe" label, the TCN network predicts the target object will move to the right in the next 6 seconds at a speed of 1.5 m / s, generating the movement trajectory key points F(10,6,4), G(11.5,6,4), H(13,6,4), I(14.5,6,4), and J(16,6,4). Using a Bezier curve to set control points (10.8,6,4) and (14.2,6,4), the smoothed trajectory key points are F(10,6,4), G'(11.4,6,4), H'(13.1,6,4), I'(14.6,6,4), and J(16,6,4). The Euclidean distance between the midpoint H'(13.1,6,4) of the trajectory and the center of the camera's field of view (10,6,4) is calculated. The trajectory deviation is 3.1 meters, exceeding the 2.5-meter deviation threshold. Correction data "Adjust horizontally to the right by 3.1 meters" is generated. The adjustment is performed in three steps using linear interpolation. The parameter adjustment dataset is as follows: Step 1 (0-2 seconds): Move 1.03 meters to the right, focal length 30 mm; Step 2 (2-4 seconds): Move 1.03 meters to the right, focal length 32 mm; Step 3 (4-6 seconds): Move 1.04 meters to the right, focal length 34 mm.

[0050] In step S14, control commands need to be generated based on the camera parameter adjustment dataset to adjust the camera operating parameters. Occlusion detection is then performed on the adjusted real-time video frame sequence to obtain occlusion data. The trajectory data is then fused to obtain real-time tracking parameters, including: Based on the camera parameters, adjust the dataset to generate control commands that conform to the camera control protocol, and drive the camera to adjust the shooting angle and focal length. For the real-time video frame sequence captured by the adjusted camera, the occlusion area features are extracted by pixel density comparison to generate occlusion detection data; If the occlusion detection data exceeds a preset occlusion threshold, the trajectory data and the occlusion detection data are fused using a linear interpolation method to generate a first tracking parameter; Based on the first tracking parameter and the abnormal emotion label, the weight of the first tracking parameter is adjusted by weighted average to obtain the real-time tracking parameter.

[0051] It should be noted that common camera control protocols include ONVIF and Pelco-D. Control commands must strictly adhere to the protocol format. For example, ONVIF angle adjustment commands must include the angle type (horizontal / vertical), adjustment direction, and angle value; focal length adjustment commands must include the target focal length value. Pixel density comparison involves calculating the pixel density (pixels per square centimeter) of the target area in the video frame and comparing it with the baseline pixel density (e.g., 150 pixels / square centimeter) when there is no occlusion. If the pixel density of a certain area is lower than 70% of the baseline value (based on statistical analysis of historical test data in similar scenarios, such as taking the average value; in practice, this can be dynamically adjusted by those skilled in the art according to the monitoring and tracking needs of different scenarios; the values ​​involved in this article are only examples), it is determined to be an occluded area. Occlusion detection data includes the occlusion location and the percentage of the occluded area (e.g., "right side of the image, occluded area percentage 30%").

[0052] The preset occlusion threshold is set based on the average value of historical test data from similar scenarios. In practice, it can be dynamically adjusted by those skilled in the art according to the monitoring and tracking needs of different scenarios. The values ​​mentioned in this article are for illustrative purposes only and are typically set to 20%. During linear interpolation fusion, the weight values ​​are set based on the target position predicted by the trajectory data and the actual occlusion situation reflected by the occlusion detection data. The first tracking parameter (such as adjusted speed and direction) is calculated with weights of 0.6 and 0.4, respectively. This can be dynamically adjusted by those skilled in the art according to the degree of occlusion. The more severe the occlusion, the higher the weight of the occlusion detection data can be (not exceeding 0.5) to prevent over-reliance on trajectory data leading to deviation. During weighted average adjustment, abnormal emotion labels are incorporated; for example, in a "tense" state, the weight of the trajectory data is increased to 0.7 to obtain the final real-time tracking parameters.

[0053] Furthermore, the abnormal emotion labels (such as "Tense - Moderate" and "Anxious - Severe") can be pre-set with different weighting factors. For example, the weighting adjustment coefficient for "Moderate" can be defined as 1.1, and for "Severe" as 1.2. When adjusting the weight of the first tracking parameter using a weighted average, the original weight of the trajectory data (e.g., 0.6) is multiplied by this coefficient, and the weight of the occlusion detection data is adjusted accordingly to maintain a total weight of 1. This aims to increase the trust in trajectory data based on behavioral trend prediction when the target's emotional abnormality is high, thereby generating real-time tracking parameters that better reflect the target's true intentions. Those skilled in the art can define the mapping relationship between emotion levels and influencing factors according to actual application scenarios.

[0054] In this step, the dataset and the ONVIF protocol used by the camera are adjusted according to the parameters to generate control commands: the first command is "move horizontally 1.25 meters to the left, adjust the focal length to 32 mm", and the second command is "move horizontally 1.25 meters to the left, adjust the focal length to 35 mm". The commands are sent to the camera control module via the network to drive the camera to perform the adjustment operation. Then, for the video frame sequence acquired after adjustment, the region of interest (ROI, such as a rectangular area of ​​300×400 pixels) where the target object is located is selected, and the pixel density of the region is calculated. If the pixel density of the ROI region in a certain frame is 105 pixels / square centimeter, which is lower than the baseline value of 150 pixels / square centimeter, and the area of ​​the region with a density lower than 70% of the baseline value (105 pixels / square centimeter) accounts for 30% of the ROI, then the occlusion detection data "Occlusion position: right side of ROI, occlusion ratio 30%" is generated.

[0055] Furthermore, judging that the occlusion ratio of 30% exceeds the preset occlusion threshold of 20%, linear interpolation is used to fuse trajectory data and occlusion detection data: the trajectory data predicts the target object's position in the next frame as (11.7, 5, 3) meters and the speed as 1.2 m / s. The occlusion detection data indicates that 30% of the right side area is occluded. The first tracking parameter is calculated with a weight of 0.6 for trajectory data and 0.4 for occlusion detection data: position (11.7 - 0.3 × 1.2 × 0.4, 5, 3) = (11.556, 5, 3) meters (where 0.3 is the 30% occlusion ratio, which is the quantitative result of occlusion interference. Multiplying by the trajectory prediction speed of 1.2 m / s can correlate the influence relationship between occlusion and target movement. Combined with the occlusion data weight of 0.4, the correction ratio can be allocated according to the reliability of the occlusion data, avoiding the deviation caused by relying solely on trajectory or occlusion data), and speed 1.2 × (1 - 0.3 × 0.4) = 1.056 m / s. Finally, combining the "tense-moderate" label, the weight of trajectory data was increased to 0.7, and the weight of occlusion detection data was reduced to 0.3. After weighted average adjustment, the real-time tracking parameters were obtained: position (11.7-0.3×1.2×0.3,5,3)=(11.608,5,3) meters, speed 1.2×(1-0.3×0.3)=1.116 meters / second.

[0056] It is worth noting that when generating control commands, the camera's adjustment accuracy must be considered, such as a horizontal movement accuracy of 0.01 meters and a focal length adjustment accuracy of 1 millimeter, to ensure that the command parameters are within the camera's controllable range. The baseline value for pixel density comparison needs to be pre-collected in an unobstructed scene, and recalibrated under different lighting conditions to avoid misjudgments caused by changes in lighting.

[0057] For example, the shopping mall camera uses the Pelco-D protocol, and the parameter adjustment dataset is "Step 1: Move 1.03 meters to the right, focal length 30 mm; Step 2: Move 1.03 meters to the right, focal length 32 mm; Step 3: Move 1.04 meters to the right, focal length 34 mm". The generated Pelco-D control commands are: Step 1: "Address 1, move 1.03 meters to the right, focal length 30 mm", Step 2: "Address 1, move 1.03 meters to the right, focal length 32 mm", Step 3: "Address 1, move 1.04 meters to the right, focal length 34 mm". The commands are sent to the camera via the RS485 interface to complete the parameter adjustment.

[0058] In the adjusted video frames, the pixel density of the target object ROI region (350×450 pixels) is 98 pixels / cm², with a baseline value of 150 pixels / cm². The occlusion area is located on the left side of the ROI, accounting for 28% of the area, generating occlusion detection data "left-side occlusion, accounting for 28%". Exceeding the 20% occlusion threshold, the trajectory data (predicted position 14.6 meters, velocity 1.5 m / s) and the occlusion detection data are linearly interpolated and fused. The first tracking parameters are obtained with weights of 0.6 and 0.4: position 14.6 - 0.28 × 1.5 × 0.4 = 14.436 meters, velocity 1.5 × (1 - 0.28 × 0.4) = 1.332 m / s. By combining the "anxiety-severe" label, the weight of the trajectory data is increased to 0.7. The adjusted real-time tracking parameters are: location 14.6-0.28×1.5×0.3=14.478 meters, speed 1.5×(1-0.28×0.3)=1.374 meters / second.

[0059] In step S15, the real-time tracking parameters need to be applied to the processing loop of the real-time video frame sequence. If frame data loss due to occlusion is detected, linear interpolation is used to fill the missing frame data and data consistency verification is performed to obtain continuous tracking data, including: The real-time tracking parameters are applied to the processing loop of the real-time video frame sequence, the shooting angle is adjusted by the camera control unit, and an optimized video frame sequence is generated; Perform frame integrity detection on the optimized video frame sequence to determine whether there are missing frames caused by occlusion; If the missing frame exists, based on the trajectory data, determine the position coordinates and morphological features of the target object in the valid frames before and after the missing frame, use linear interpolation to calculate the key feature parameters of the target object in the missing frame, fill the missing frame data, and obtain the filled video frame sequence. The data consistency of the padded video frame sequence is checked, and the positional deviation and morphological feature variation of the target object between adjacent frames are calculated. If the change in position deviation and morphological features exceeds a preset continuity threshold, the key feature parameters are re-optimized based on the trajectory data until the preset continuity threshold is met, thereby generating continuous tracking data.

[0060] It should be noted that when real-time tracking parameters are applied to loop processing, the camera control unit adjusts the shooting angle (horizontal and vertical angles) in real time based on the position and speed information in the parameters to ensure that the target object is always centered in the frame. The acquired video frame sequence is the optimized video frame sequence. Frame integrity detection compares the integrity of key areas (face, torso) of the target object in adjacent frames. If the proportion of missing key areas in a frame exceeds 30% (set based on the average value of historical test data of similar scenarios; in practice, it can be dynamically adjusted by those skilled in the art according to the monitoring and tracking needs of different scenarios; the values ​​involved in this article are only examples), it is determined to be a missing frame.

[0061] Morphological features include limb joint angles (such as elbow angle and shoulder angle) and contour dimensions; key feature parameters are position coordinates and quantified values ​​of morphological features. During linear interpolation filling, the parameters of the missing frame are calculated proportionally to the time interval, based on the key feature parameters of the valid frames before and after the missing frame. For example, if the missing frame is located in... (Location )and (Location )between, - = - Then the missing frame position =( + ) / 2. The continuity threshold for data consistency verification is set by experts in the field based on statistical analysis of historical test data from similar scenarios, such as taking the average value. In practice, it can be dynamically adjusted by those skilled in the art according to the monitoring and tracking needs of different scenarios. The values ​​involved in this article are only examples. For example, the position deviation is set to no more than 10 pixels and the change in limb joint angle is set to no more than 15°. If these are exceeded, the key feature parameters are recalculated until the threshold is met. The final generated continuous tracking data contains a complete and continuous video frame sequence and the corresponding feature parameters. The continuity threshold can be adaptively adjusted according to the video frame rate and the target motion speed. For high-speed motion scenarios, the threshold should be appropriately increased; for low-speed fine tracking scenarios, the threshold can be decreased.

[0062] In this step, the real-time tracking parameters (position 11.608 meters, speed 1.116 meters / second) are first input into the video frame processing loop. The camera control unit adjusts the horizontal angle to 15° according to the parameters to ensure that the target object is centered in the frame, and an optimized video frame sequence (frame rate 25fps) is generated. Then, the optimized video frame sequence is checked for frame integrity. In frame 120, the target object's face is obscured by other customers, with a missing rate of 45%, and is determined to be a missing frame. Next, the position coordinates of the previous valid frame (frame 119) and the elbow angle of 90° are determined, and the position coordinates of the next valid frame (frame 121) and the elbow angle of 100° are determined.

[0063] Furthermore, linear interpolation was used to calculate the key feature parameters of frame 120: position (11.5+11.7) / 2=11.6 meters, elbow angle (90+100) / 2=95°. These parameters were used to fill in the data for frame 120, resulting in the filled video frame sequence. Then, data consistency was checked, and the positional deviation between frame 119 and frame 120 was calculated. =0.1 meters (approximately 2 pixels), less than the 10-pixel threshold; elbow angle change 95-90=5°, less than the 15° threshold; the positional deviation of the 120th frame from the 121st frame is 0.1 meters and the angle change is 5°, which also meets the threshold, generating continuous tracking data containing the complete frame sequence.

[0064] It is worth noting that frame integrity detection can be combined with the occlusion detection data from step S14. If the occlusion rate of a frame exceeds 50% and key areas are occluded, it can be directly identified as a missing frame, improving detection efficiency. When linear interpolation fills multiple consecutive missing frames (e.g., 3 frames), the number of reference frames needs to be increased (e.g., 2 frames before and after), and multiple interpolations should be used to ensure filling accuracy. If the data consistency check still does not meet the threshold after multiple adjustments, it is necessary to return to step S13 to re-optimize the trajectory data to avoid abnormal filling data due to trajectory deviation.

[0065] For example, in the optimized video frame sequence of a shopping mall surveillance camera, frames 200-202 are missing frames (the target is obscured by a pillar). Frame 199 (a valid frame) has a position coordinate of 14.4 meters and a shoulder angle of 180°; frame 203 (a valid frame) has a position coordinate of 14.8 meters and a shoulder angle of 170°. Linear interpolation is used to fill the gaps: frame 200 has a position coordinate of (14.4+14.8) / 4×1+14.4=14.5 meters and a shoulder angle of (180-170) / 4×1+170=177.5°; frame 201 has a position coordinate of 14.6 meters and a shoulder angle of 175°; frame 202 has a position coordinate of 14.7 meters and a shoulder angle of 172.5°. During verification, the position deviation of 0.1 meters (2 pixels) and the angle change of 2.5° in frames 199-200, and the deviation of 0.1 meters and the angle change of 2.5° in frames 200-201, both meet the threshold and generate continuous tracking data.

[0066] In step S16, the processing flow of the real-time video frame sequence needs to be updated based on the continuous tracking data. The target object features are re-extracted and the state is updated. If it is still a preset abnormal state, new behavioral change trend data is generated and time-series analysis is performed to obtain the final tracking logic sequence, including: The processing flow based on the continuous tracking data updates the real-time video frame sequence, and the updated facial expression features and posture features are re-extracted from the real-time video frame sequence; Based on the updated facial expression and posture features, it is determined whether the updated state of the target object belongs to a preset abnormal state. If the updated state is still a preset abnormal state, new behavioral change trend data is generated. The new behavioral change trend data is analyzed over time to generate abnormal emotion tracking logic, resulting in the final tracking logic sequence.

[0067] It should be noted that the update process includes adjusting the video frame rate (e.g., increasing it from 25fps to 30fps when the target moves quickly) and optimizing the region of interest (ROI) size to ensure more accurate capture of target features. When re-extracting features, the OpenPose algorithm from step S11 and the VGG-Face model from step S12 are used to obtain updated facial features (e.g., eyebrow raising amplitude of 4.8 mm, eye squinting frequency of 2.8 times / second) and posture features (e.g., arm swinging frequency of 1.5 times / second). The updated state judgment still uses the original preset threshold. If the abnormal conditions are still met, new behavioral change trend data is generated using the weighted fusion method of step S12. Temporal analysis uses a sliding window (e.g., a 5-second window) to analyze the changing patterns of the new behavioral change trend data. For example, a continuous increase in data indicates intensified emotion, and tracking logic is generated accordingly (e.g., "adjusting camera parameters every 0.5 seconds" and "increasing the occlusion detection frequency to once per frame"), arranged chronologically to form the final tracking logic sequence.

[0068] The time-series analysis can employ a sliding window mechanism to calculate statistical characteristics such as mean, variance, and slope of the behavioral change trend data within the window. If the slope remains positive and exceeds a set threshold, it is determined that the emotion has intensified, and an instruction to increase the tracking frequency is generated. If the data fluctuates significantly, an instruction to initiate occlusion detection is generated. Those skilled in the art can design corresponding state machines or rule sets based on specific business logic to map data trends into specific camera control logic sequences.

[0069] In this step, firstly, based on continuous tracking data, the video frame rate is increased from 25fps to 30fps, and the ROI area is expanded from 300×400 pixels to 350×450 pixels, updating the processing flow of the real-time video frame sequence. Then, frames 125-135 are selected from the updated frame sequence. In another implementation, posture features (arm swing frequency 1.5 times / second, elbow angle change range 80-100°) can be extracted using the OpenPose algorithm, and facial expression features (eyebrow raising amplitude 4.8 mm, eye squinting frequency 2.8 times / second) can be extracted using the VGG-Face model to obtain updated feature data. Next, the updated facial expression features are compared with preset amplitude and frequency thresholds. If 4.8 mm is greater than 4 mm and 2.8 times / second is greater than 2 times / second, the updated state is still determined to be "tense-moderate".

[0070] Furthermore, the facial expression features and posture features were quantified separately. The eyebrow raising amplitude of 4.8 mm was quantized as 4.8 (4.8 ÷ 10 × 10 = 4.8); the eye squinting frequency of 2.8 times / second was quantized as 5.6 (2.8 ÷ 5 × 10 = 5.6), resulting in the quantized facial expression features [4.8, 5.6]; the arm swing frequency was mapped according to "0-3 times / second corresponds to 0-10" (3 times / second is the threshold value for abnormal swinging of the target in a crowded scene, exceeding which is judged as abnormal behavior), and 1.5 times / second was quantized as 5 (1.5 ÷ 3 × 10 = 5); the elbow angle was taken as the average value of 80-100°, 90°, and mapped according to "60-120° corresponds to 0-10" (60° is the angle of the arm hanging down naturally, and 120° is the threshold angle of abnormal swinging), and 90° was quantized as 5 ((90-60) ÷ (120-60) × 10 = 5), resulting in the quantized posture features [5, 5]. The values ​​set in the quantization mapping rule based on the abnormal posture judgment standard can be modified and adjusted by those skilled in the art according to different actual application scenarios, so that the quantized results are more in line with actual needs.

[0071] Furthermore, the quantized facial expression features [4.8, 5.6] and quantized posture features [5, 5] were fused with weights of 0.6 and 0.4 respectively to generate new behavioral trend data [4.8×0.6+5×0.4, 5.6×0.6+5×0.4]=[4.88, 5.36]. Finally, a 5-second sliding window was used to perform time-series analysis on the new data. It was found that within 10 seconds, the eyebrow raising amplitude increased from 4.8 mm to 5.2 mm, and the arm swing frequency increased from 1.5 times / second to 1.8 times / second, indicating an increase in tension. The final tracking logic sequence was generated and marked as "high tension state".

[0072] It's worth noting that the processing flow needs to be dynamically adjusted based on the target object's motion state. When the motion is slow, the frame rate is reduced to decrease computational resource consumption; when the motion is fast, the frame rate is increased to ensure complete feature capture. The frequency of feature re-extraction can be set based on state stability: once every 5 seconds when the state is stable, and once every 2 seconds when the state fluctuates. The sliding window size for time-series analysis can be adjusted according to scenario requirements. For security scenarios requiring higher sensitivity, the window can be set to 3 seconds; for general monitoring scenarios, the window can be set to 8 seconds.

[0073] For example, in shopping mall surveillance, continuous tracking data shows that the target object's movement speed is increasing. The frame rate is increased to 30fps, and the ROI is expanded to 400×500 pixels. The re-extracted facial features are an eyebrow raise of 5.5 mm and an eye squinting frequency of 3.2 times / second. Postural features are an arm swing frequency of 2 times / second and a gait frequency of 2.5 steps / second. The updated state is "Anxiety - Severe". New behavioral change trend data is generated by fusing these features (5.5×0.6+2×0.4,3.2×0.6+2.5×0.4)=(4.1,3.12). Temporal analysis reveals a continuous increase in data within 5 seconds, generating the final tracking logic sequence, such as "Severe Anxiety State".

[0074] It should be noted that the various thresholds, weights, and specific values ​​(such as amplitude threshold, frequency threshold, deviation threshold, weighted fusion weight, occlusion threshold, continuity threshold, etc.) involved in this invention are all derived from preferred embodiments under specific experimental conditions and have undergone extensive testing to verify their effectiveness. The setting of these values ​​comprehensively considers algorithm performance, hardware accuracy, statistical characteristics of common application scenarios (such as shopping malls and train stations), and general laws of human behavior. Those skilled in the art will understand that these parameters are not the only implementation method of this invention. In practical applications, these parameters can be optimized and adjusted through limited conventional experiments based on specific camera resolution, processing chip computing power, ambient lighting conditions, and the specific needs of the target behavior. Such adjustments do not require creative effort. The core of the technical solution of this invention lies in the system architecture, processing flow, and algorithm combination, rather than the specific parameter values.

[0075] In summary, this invention discloses a machine vision-based automatic camera tracking method, comprising: acquiring a real-time video frame sequence; performing image preprocessing to obtain an initial image set containing features of the target object; performing feature recognition on the initial image set to extract facial expression and posture features of the target object and analyze its state; if the state is determined to be a preset abnormal state, performing data fusion on the initial image set to generate behavioral change trend data; based on the behavioral change trend data, performing trajectory prediction and smoothing processing to determine trajectory data; calculating the trajectory deviation by combining the current position parameters of the camera to obtain a camera parameter adjustment dataset; and adjusting the camera parameter data accordingly. The system generates control commands to adjust camera operating parameters and simultaneously performs occlusion detection on the real-time video frame sequence to obtain occlusion data. It then fuses the trajectory data to obtain real-time tracking parameters. These parameters are applied to the processing loop of the real-time video frame sequence. If frame data loss due to occlusion is detected, linear interpolation is used to fill the missing frame data, and data consistency verification is performed to obtain continuous tracking data. Based on the continuous tracking data, the processing flow of the real-time video frame sequence is updated, target object features are re-extracted, and the state is updated. If the state is still a preset abnormal state, new behavioral change trend data is generated and time-series analysis is performed to obtain the final tracking logic sequence.

[0076] This invention employs a preprocessing workflow of "video frame acquisition - target separation - noise removal - contrast enhancement - key point extraction," combined with face detection and median filtering techniques to filter redundant interference, providing high-quality facial expression and posture feature data for subsequent analysis. This addresses the problem of existing technologies where target feature extraction is easily affected by background. Simultaneously, this invention combines facial expression recognition with body posture analysis. It uses convolutional neural networks to extract facial features to determine abnormal emotions, fuses limb key point displacement vectors to generate behavioral trend data, and uses emotion as a trajectory prediction variable to reduce prediction bias caused by sudden behaviors induced by emotions, thus improving the accuracy of behavior prediction. Furthermore, this invention uses pixel density comparison to identify occluded areas and combines trajectory data with linear interpolation to fill in missing occluded frames. Data consistency verification ensures continuity, avoiding tracking interruptions caused by occlusion in existing technologies and ensuring continuous target locking. In addition, this invention calculates camera parameter adjustment values ​​based on behavioral trend data, integrates occlusion detection to optimize tracking parameters, forming a dynamic adjustment mechanism. This mechanism can adjust parameters in real time according to target behavior and environmental occlusion, solving the problems of fixed parameters and poor adaptability in existing technologies, and ensuring stable tracking in congested environments.

[0077] Reference Figure 2 The second embodiment of the present invention provides a camera automatic tracking device based on machine vision, comprising: The data acquisition module acquires real-time video frame sequences, performs image preprocessing, and obtains an initial image set containing the features of the target object. The feature analysis module performs feature recognition on the initial image set, extracts the facial expression features and posture features of the target object and analyzes the state. If the state is determined to be a preset abnormal state, the initial image set is fused to generate behavioral change trend data. The trajectory prediction module performs trajectory prediction and smoothing processing based on the behavior change trend data, determines the trajectory data, calculates the trajectory deviation by combining the current position parameters of the camera, and obtains the camera parameter adjustment dataset. The control and tracking module generates control commands based on the camera parameter adjustment dataset, adjusts the camera operating parameters, and simultaneously performs occlusion detection on the real-time video frame sequence to obtain occlusion data. It then fuses the trajectory data to obtain real-time tracking parameters. The frame loss compensation module applies the real-time tracking parameters to the processing loop of the real-time video frame sequence. If frame data loss due to occlusion is detected, linear interpolation is used to fill the missing frame data and data consistency verification is performed to obtain continuous tracking data. The logic output module updates the real-time video frame sequence based on the continuous tracking data, re-extracts the target object features and updates the state. If it is still a preset abnormal state, it generates new behavior change trend data and performs time series analysis to obtain the final tracking logic sequence.

[0078] It should be noted that the machine vision-based automatic camera tracking device provided in this embodiment of the invention is used to execute all the process steps of the machine vision-based automatic camera tracking method in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0079] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0080] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A machine vision-based automatic camera tracking method, characterized in that, include: Acquire real-time video frame sequences, perform image preprocessing, and obtain an initial image set containing the features of the target object; Feature recognition is performed on the initial image set to extract the facial expression and posture features of the target object and analyze its state. If the state is determined to be a preset abnormal state, data fusion is performed on the initial image set to generate behavioral change trend data. Based on the behavioral change trend data, trajectory prediction and smoothing are performed to determine the trajectory data. The trajectory deviation is calculated by combining the current position parameters of the camera to obtain the camera parameter adjustment dataset. Based on the camera parameters, the dataset is adjusted to generate control commands, which adjust the camera's operating parameters. At the same time, occlusion detection is performed on the real-time video frame sequence to obtain occlusion data. The trajectory data is then fused to obtain real-time tracking parameters. The real-time tracking parameters are applied to the processing loop of the real-time video frame sequence. If frame data loss due to occlusion is detected, linear interpolation is used to fill the missing frame data and data consistency is checked to obtain continuous tracking data. The processing flow based on the continuous tracking data updates the real-time video frame sequence, re-extracts the target object features and updates the state. If it is still a preset abnormal state, new behavior change trend data is generated and time series analysis is performed to obtain the final tracking logic sequence.

2. The automatic camera tracking method based on machine vision according to claim 1, characterized in that, The step of acquiring a real-time video frame sequence, performing image preprocessing, and obtaining an initial image set containing the features of the target object includes: Acquire the real-time video frame sequence captured by the camera; Image data of the target person is obtained from the real-time video frame sequence, and first image data containing the features of the target object is separated by face detection and human body contour segmentation. For the first image data, background noise is removed by median filtering to generate the second image data; For the second image data, histogram equalization is used to enhance image contrast, generating the third image data; For the third image data, facial expression features and body posture features are extracted through key point detection to generate initial facial images and initial body images, forming an initial image set.

3. The automatic camera tracking method based on machine vision according to claim 1, characterized in that, The initial image set is subjected to feature recognition to extract facial expression and posture features of the target object and analyze its state. If the state is determined to be a preset abnormal state, the initial image set is fused to generate behavioral change trend data, including: For the initial facial images in the initial image set, a convolutional neural network is used to extract facial expression features, analyze the degree of eyebrow raising and the frequency of eye squinting, and generate expression feature data. For the facial expression feature data, if the eyebrow raising amplitude is greater than a preset amplitude threshold or the eye squinting frequency exceeds a preset frequency threshold, it is judged as a preset abnormal state and an abnormal emotion label is generated. Based on the abnormal emotion label, the human posture key point sequence is extracted from the initial limb images in the initial image set through key point detection, the displacement vector between key points is calculated, and displacement vector data is generated. The displacement vector data is combined with the facial expression feature data and weighted fusion is used to generate behavioral change trend data.

4. The automatic camera tracking method based on machine vision according to claim 3, characterized in that, Based on the behavioral change trend data, trajectory prediction and smoothing are performed to determine the trajectory data. The trajectory deviation is calculated by combining the current camera position parameters to obtain a camera parameter adjustment dataset, including: Obtain the current position parameters of the camera; The dynamic features of the behavioral change trend data are extracted using a convolutional neural network, and the abnormal emotion labels are combined to predict the trajectory of the target object. The movement trajectory is smoothed using a Bézier curve algorithm to determine the trajectory data; The deviation between the trajectory data and the current position parameters of the camera is calculated. If the deviation exceeds a preset deviation threshold, the trajectory deviation is obtained by calculating the Euclidean distance, and correction data is generated. Based on the correction data, linear interpolation is used to calculate the adjustment values ​​of the camera translation and scaling parameters, generating a camera parameter adjustment dataset.

5. The automatic camera tracking method based on machine vision according to claim 4, characterized in that, The step of generating control commands based on the camera parameter adjustment dataset, adjusting the camera operating parameters, performing occlusion detection on the real-time video frame sequence after parameter adjustment to obtain occlusion data, and fusing the trajectory data to obtain real-time tracking parameters includes: Based on the camera parameters, adjust the dataset to generate control commands that conform to the camera control protocol, and drive the camera to adjust the shooting angle and focal length. For the real-time video frame sequence captured by the adjusted camera, the occlusion area features are extracted by pixel density comparison to generate occlusion detection data; If the occlusion detection data exceeds a preset occlusion threshold, the trajectory data and the occlusion detection data are fused using a linear interpolation method to generate a first tracking parameter; Based on the first tracking parameter and the abnormal emotion label, the weight of the first tracking parameter is adjusted by weighted average to obtain the real-time tracking parameter.

6. The automatic camera tracking method based on machine vision according to claim 1, characterized in that, The process loop that applies the real-time tracking parameters to the real-time video frame sequence includes, if frame data loss due to occlusion is detected, linear interpolation is used to fill the missing frame data and data consistency verification is performed to obtain continuous tracking data, including: The real-time tracking parameters are applied to the processing loop of the real-time video frame sequence, the shooting angle is adjusted by the camera control unit, and an optimized video frame sequence is generated; Perform frame integrity detection on the optimized video frame sequence to determine whether there are missing frames caused by occlusion; If the missing frame exists, based on the trajectory data, determine the position coordinates and morphological features of the target object in the valid frames before and after the missing frame, use linear interpolation to calculate the key feature parameters of the target object in the missing frame, fill the missing frame data, and obtain the filled video frame sequence. The data consistency of the padded video frame sequence is checked, and the positional deviation and morphological feature variation of the target object between adjacent frames are calculated. If the change in position deviation and morphological features exceeds a preset continuity threshold, the key feature parameters are re-optimized based on the trajectory data until the preset continuity threshold is met, thereby generating continuous tracking data.

7. The automatic camera tracking method based on machine vision according to claim 1, characterized in that, The process of updating the real-time video frame sequence based on the continuous tracking data involves re-extracting the target object features and updating the state. If the state is still a preset abnormal state, new behavioral change trend data is generated and time-series analysis is performed to obtain the final tracking logic sequence, including: The processing flow based on the continuous tracking data updates the real-time video frame sequence, and the updated facial expression features and posture features are re-extracted from the real-time video frame sequence; Based on the updated facial expression and posture features, it is determined whether the updated state of the target object belongs to a preset abnormal state. If the updated state is still a preset abnormal state, new behavioral change trend data is generated. The new behavioral change trend data is analyzed over time to generate abnormal emotion tracking logic, resulting in the final tracking logic sequence.

8. A camera-based automatic tracking device based on machine vision, characterized in that, include: The data acquisition module acquires real-time video frame sequences, performs image preprocessing, and obtains an initial image set containing the features of the target object. The feature analysis module performs feature recognition on the initial image set, extracts the facial expression features and posture features of the target object and analyzes the state. If the state is determined to be a preset abnormal state, the initial image set is fused to generate behavioral change trend data. The trajectory prediction module performs trajectory prediction and smoothing processing based on the behavior change trend data, determines the trajectory data, calculates the trajectory deviation by combining the current position parameters of the camera, and obtains the camera parameter adjustment dataset. The control and tracking module generates control commands based on the camera parameter adjustment dataset, adjusts the camera operating parameters, and simultaneously performs occlusion detection on the real-time video frame sequence to obtain occlusion data. It then fuses the trajectory data to obtain real-time tracking parameters. The frame loss compensation module applies the real-time tracking parameters to the processing loop of the real-time video frame sequence. If frame data loss due to occlusion is detected, linear interpolation is used to fill the missing frame data and data consistency verification is performed to obtain continuous tracking data. The logic output module updates the real-time video frame sequence based on the continuous tracking data, re-extracts the target object features and updates the state. If it is still a preset abnormal state, it generates new behavior change trend data and performs time series analysis to obtain the final tracking logic sequence.

Images