Intelligent camera positioning identification method and system and intelligent glasses

Abstract

The invention relates to the technical field of camera positioning and recognition, in particular to an intelligent camera positioning and recognition method and system and intelligent glasses. The method comprises the following steps: acquiring an original image sequence of a camera; performing multi-band decoupling on the original image sequence of the camera to obtain multi-band component data of the image sequence; performing low-frequency target motion component identification on the image sequence multi-band component data to obtain low-frequency target motion component data; obtaining IMU inertial data of an operator; performing gait cycle recognition on the IMU inertial data of the operator to obtain gait phase feature data; and performing variable gain image stabilization compensation on the original image sequence of the camera according to the gait phase feature data to obtain an image stabilization compensation image sequence. According to the invention, the problem of image blurring caused by the human gait cycle can be effectively reduced, and the definition and stability of the image are improved.

Description

Technical Field

[0001] The present invention relates to the field of camera positioning and recognition technology, and in particular to an intelligent camera positioning and recognition method, system and smart glasses. Background Art

[0002] In the field of intelligent camera technology, users wearing smart glasses can perform image recognition and tracking. Existing visual positioning systems face unique dynamic perception challenges. When a user walks rapidly and frequently changes their viewing angle, the image sequences captured by the wearable camera will produce non-uniform motion blur. This periodic blurring, caused by the human gait cycle, couples with the autonomous motion of the target envelope, leading to feature confusion when traditional optical flow algorithms calculate motion vectors. Conventional edge detection methods, especially when processing standardized products with similar surface textures, have difficulty distinguishing between pseudo-motion caused by the user's head shaking and actual displacement of the target object. Existing solutions generally ignore high-frequency jitter interference caused by micro-movements of the user's neck. Experimental data shows that when the wearer performs movements, tremors in the neck muscles can cause the camera to produce periodic jitter with an amplitude of approximately 0.5° and a frequency of 8-12Hz. The amplification effect of this micro-motion in telephoto mode can cause resonant shifts in the target bounding box. Summary of the Invention

[0003] Based on this, it is necessary for the present invention to provide an intelligent camera positioning and recognition method, system and smart glasses to solve at least one of the above technical problems.

[0004] To achieve the above object, a method for positioning and identifying an intelligent camera includes the following steps:

[0005] Step S1: obtaining a camera original image sequence; performing multi-band decoupling on the camera original image sequence to obtain multi-band component data of the image sequence; performing low-frequency target motion component identification on the multi-band component data of the image sequence to obtain low-frequency target motion component data;

[0006] Step S2: Acquire the operator's IMU inertial data; perform gait cycle recognition on the operator's IMU inertial data to obtain gait phase feature data; perform variable gain stabilization compensation on the camera's original image sequence based on the gait phase feature data to obtain a stabilization-compensated image sequence;

[0007] Step S3: acquiring multimodal operating environment data; estimating the operator's motion state based on the multimodal operating environment data to obtain operator motion state data; calculating the target object optical flow on the image stabilization compensation image sequence based on the operator motion state data to obtain a target motion vector field;

[0008] Step S4: motion trajectory evaluation is performed on the target motion vector field to obtain a target trajectory candidate point set; RANSAC outlier elimination is performed on the target trajectory candidate point set to obtain an effective target trajectory; physical constraint verification is performed on the effective target trajectory to obtain a target real motion trajectory;

[0009] Step S5: target object multi-feature extraction is performed based on the target real motion trajectory to obtain a target comprehensive feature vector; multi-target fusion tracking is performed on the target object according to the target comprehensive feature vector to obtain a target positioning and recognition result.

[0010] The present application can effectively separate different frequency components in the image sequence through multi-band decoupling and low-frequency target motion component identification, thereby accurately extracting the low-frequency motion information of the target object, solving the feature confusion problem caused by the coupling of image blur and target autonomous motion due to the human gait cycle in the prior art. At the same time, the gait cycle is identified in combination with the IMU inertial data of the operator, and the original image sequence of the camera is compensated for variable gain image stabilization compensation accordingly, which can significantly reduce the high-frequency jitter interference caused by the operator's head shaking and neck micro-motion, and avoid the resonance type deviation of the target bounding box in the long focal mode. In addition, the operator's motion state is estimated based on the multi-modal operating environment data, and the target object optical flow is calculated accordingly, which can more accurately reflect the real motion state of the target object, and improve the positioning and tracking accuracy in complex dynamic environments. Through motion trajectory evaluation, outlier elimination and physical constraint verification of the target motion vector field, the accuracy and reliability of the target trajectory are further ensured, and finally multi-feature extraction and fusion tracking are performed based on the target real motion trajectory, realizing accurate positioning and recognition of the target object, and effectively improving the efficiency and accuracy of dynamic sorting operation in the logistics and warehousing center.

[0011] Preferably, the present application provides an intelligent camera positioning and recognition system for executing the intelligent camera positioning and recognition method as described above, which comprises:

[0012] An image preprocessing module is configured to acquire the original image sequence of the camera; perform multi-band decoupling on the original image sequence of the camera to obtain image sequence multi-band component data; and perform low-frequency target motion component identification on the image sequence multi-band component data to obtain low-frequency target motion component data.

[0013] An image stabilization compensation module is configured to acquire the operator IMU inertial data; perform gait cycle identification on the operator IMU inertial data to obtain gait phase feature data; and perform variable gain image stabilization compensation on the original image sequence of the camera according to the gait phase feature data to obtain an image stabilization compensation image sequence.

[0014] a motion estimation module, configured to acquire multi-modal operating environment data, perform motion state estimation on the operator based on the multi-modal operating environment data, and obtain operator motion state data, and perform target object optical flow calculation on the stabilized image sequence according to the operator motion state data, and obtain a target motion vector field;

[0015] a trajectory evaluation and screening module, configured to perform motion trajectory evaluation on the target motion vector field, obtain a target trajectory candidate point set, perform RANSAC outlier elimination on the target trajectory candidate point set, obtain an effective target trajectory, and perform physical constraint verification on the effective target trajectory, and obtain a target real motion trajectory;

[0016] a target recognition and tracking module, configured to perform target object multi-feature extraction based on the target real motion trajectory, obtain a target comprehensive feature vector, and perform multi-target fusion tracking on the target object according to the target comprehensive feature vector, and obtain a target positioning recognition result.

[0017] In the present application, the image preprocessing module effectively separates different frequency components in the image by performing multi-band decoupling and low-frequency target motion component identification on the camera original image sequence, and accurately extracts the low-frequency motion information of the target object, solving the feature confusion problem caused by the coupling of image blur and target autonomous motion due to the operator's gait cycle. The image stabilization compensation module combines the IMU inertial data of the operator to identify the gait cycle, and accordingly performs variable gain image stabilization compensation on the camera original image sequence, significantly reducing the high-frequency jitter interference caused by the operator's head shaking and neck micro-motion, avoiding the resonance type deviation of the target bounding box in the long focus mode, thereby improving the stability and clarity of the image. The motion estimation module estimates the motion state of the operator by acquiring multi-modal operating environment data, and accordingly performs target object optical flow calculation, which can more accurately reflect the real motion state of the target object, even in a complex dynamic environment, effectively improving the accuracy of target positioning and tracking, and enhancing the adaptability and robustness of the system. The trajectory evaluation and screening module performs motion trajectory evaluation on the target motion vector field, eliminates abnormal points, and performs physical constraint verification, further ensuring the accuracy and reliability of the target trajectory, avoiding false judgments caused by noise or interference, and thereby providing more accurate trajectory information for the final target recognition. The target recognition and tracking module performs multi-feature extraction and fusion tracking based on the target real motion trajectory, realizes accurate positioning and recognition of the target object, and through the comprehensive use of multiple feature vectors, the system can effectively meet the target recognition requirements in complex scenes, improving the accuracy and efficiency of recognition.

[0018] Preferably, the present application further provides a smart glasses, comprising: a processor, a memory, a camera, a display screen, a communication circuit, an audio component and a ranging component; the camera comprises a visible light camera and an infrared camera; the memory stores a program capable of being loaded and executed by the processor to perform the smart camera positioning and identification method as described above. BRIEF DESCRIPTION OF DRAWINGS

[0019] Other features, objects, and advantages of the application will become more apparent from the following detailed description when read in connection with the following drawings:

[0020] Fig. 1 A step flow diagram of the smart camera positioning and identification method of an embodiment is shown.

[0021] Fig. 2 A detailed step flow diagram of step S27 of an embodiment is shown.

[0022] Fig. 3 A detailed step flow diagram of step S37 of an embodiment is shown. DETAILED DESCRIPTION

[0023] The technical method of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of the present application.

[0024] In addition, the accompanying drawings are only schematic and are not necessarily drawn to scale. Like reference numerals designate like parts throughout the several views and the use of "first" and "second" nomenclature to designate a component in different figures is used only for purposes of differentiation and does not connote any order, sequence or importance. The accompanying drawings illustrate various embodiments of the application and together with the general description given above and the detailed description below, serve to explain the principles of the application. In particular, the drawings show one or more embodiments of the application and the operation thereof.

[0025] It should be understood that, although the terms "first", "second" and the like can be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of example embodiments. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0026] To achieve the above-mentioned purposes, please refer toFigs. 1 to 3 The application provides a smart camera positioning and recognition method, comprising the following steps:

[0027] Step S1: acquiring a camera original image sequence; performing multi-band decoupling on the camera original image sequence to obtain image sequence multi-band component data; performing low-frequency target motion component recognition on the image sequence multi-band component data to obtain low-frequency target motion component data;

[0028] Step S2: acquiring operator IMU inertial data; performing gait cycle recognition on the operator IMU inertial data to obtain gait phase feature data; performing variable gain image stabilization compensation on the camera original image sequence according to the gait phase feature data to obtain a stabilized image sequence;

[0029] Step S3: acquiring multi-modal operating environment data; performing motion state estimation on the operator based on the multi-modal operating environment data to obtain operator motion state data; performing target object optical flow calculation on the stabilized image sequence according to the operator motion state data to obtain a target motion vector field;

[0030] Step S4: performing motion trajectory evaluation on the target motion vector field to obtain a target trajectory candidate point set; performing RANSAC outlier rejection on the target trajectory candidate point set to obtain an effective target trajectory; performing physical constraint verification on the effective target trajectory to obtain a target real motion trajectory;

[0031] Step S5: performing target object multi-feature extraction based on the target real motion trajectory to obtain a target comprehensive feature vector; performing multi-target fusion tracking on the target object according to the target comprehensive feature vector to obtain a target positioning and recognition result.

[0032] Preferably, step S1 comprises the following steps:

[0033] Step S11: acquiring a camera original image sequence; performing pre-processing filtering on each image in the camera original image sequence to obtain a denoised camera image sequence;

[0034] Step S12: performing continuous frame difference calculation on the denoised camera image sequence to obtain inter-frame motion difference value data;

[0035] Step S13: performing time domain sampling and recombination on the denoised camera image sequence based on the inter-frame motion difference value data to obtain an image sequence motion time sequence matrix;

[0036] Step S14: performing fast Fourier transform on the image sequence motion time sequence matrix to obtain image sequence frequency spectrum data;

[0037] Step S15: performing frequency spectrum energy density calculation on the image sequence frequency spectrum data to obtain image frequency spectrum energy distribution features;

[0038] Step S16: performing band separation on the image sequence spectral data according to the image spectral energy distribution characteristics, to obtain image sequence multi-band component data;

[0039] Step S17: performing low-frequency target motion component identification on the image sequence multi-band component data, to obtain low-frequency target motion component data.

[0040] In this embodiment, in the dynamic sorting scenario of the logistics warehouse center, the operator wears smart glasses for package recognition and tracking. The visible light camera on the smart glasses continuously collects raw image sequences at a frequency of 30 frames per second. A Gaussian filter function in the Open CV library is used to pre-process and filter each image. The size of the Gaussian kernel is set to 5x5 and the standard deviation is set to 1.5. Through Gaussian filtering, random noise in the image can be effectively smoothed, and a denoised camera image sequence is obtained. After obtaining the denoised camera image sequence, the image processing toolbox in MATLAB software is used to calculate the difference between consecutive frames. For each pair of adjacent frames in the sequence, the pixel value of the latter frame image is subtracted from the corresponding pixel value of the former frame, thereby obtaining the inter-frame motion difference value data. This process can highlight the changing part of the image sequence due to object motion. For example, when the operator walks between shelves, the motion of the package will be clearly displayed in the inter-frame difference value data. Based on the obtained inter-frame motion difference value data, the NumPy library in Python programming language is used to perform time domain sampling and reorganization on the denoised camera image sequence. According to a certain sampling interval (for example, sampling once every 2 frames), the inter-frame motion difference value data is rearranged and combined to form an image sequence motion time sequence matrix. This matrix can more compactly represent the time variation of motion information in the image sequence. For example, through this time domain sampling and reorganization, the continuous image motion changes can be presented in the form of a matrix. The image sequence motion time sequence matrix is processed using the Fast Fourier Transform (FFT) function in MATLAB. The FFT operation is performed on each row (representing a time sequence) in the matrix to obtain image sequence frequency spectrum data. Through FFT, the motion information in the image sequence can be decomposed into components of different frequencies. For example, the low-frequency motion component caused by the operator's walking gait and the existing high-frequency jitter component can be clearly seen. After obtaining the image sequence frequency spectrum data, the signal processing toolbox in MATLAB is used to calculate the spectral energy density. The amplitude of each frequency point of the spectral data is squared and then normalized to obtain the image spectral energy distribution feature. This feature can intuitively reflect the energy proportion of different frequency components in the image sequence motion. For example, by analyzing the energy distribution feature, it can be found that the energy of the low-frequency band is mainly concentrated around a certain specific frequency, which corresponds to the operator's walking gait frequency, while the energy of the high-frequency band is relatively scattered, which is related to the camera jitter. According to the obtained image spectral energy distribution feature, the signal processing module in the SciPy library in Python is used to separate the image sequence frequency spectrum data into frequency bands.According to the energy distribution characteristics, different frequency band ranges are determined, for example, 0-2Hz is defined as a low frequency band (mainly containing autonomous motion of the target object and motion caused by the operator walking), 2-10Hz is defined as a medium frequency band (containing some small actions of the operator or environmental interference), and 10Hz or more is defined as a high frequency band (mainly high-frequency jitter of the camera). The frequency spectrum data is filtered by a band-pass filter respectively, and component data of different frequency bands is extracted, so as to obtain multi-frequency band component data of the image sequence. For details of the implementation process of step S17, please refer to the sub-steps of step S17.

[0041] Especially important is that step S17 further includes the following steps:

[0042] Step S171: performing motion source identification on the multi-frequency band component data of the image sequence to obtain image motion source identification data;

[0043] Step S172: decoupling and separating the multi-frequency band component data of the image sequence based on the image motion source identification data to obtain high-frequency jitter component data;

[0044] Step S173: performing jitter intensity evaluation on the high-frequency jitter component data to obtain high-frequency jitter intensity evaluation parameters; and performing band-stop filtering configuration on the high-frequency jitter component data according to the high-frequency jitter intensity evaluation parameters to obtain jitter data filter configuration parameters;

[0045] Step S174: performing band-stop filtering on the high-frequency jitter component data based on the jitter data filter configuration parameters to obtain medium-frequency gait component data;

[0046] Step S175: selecting a wavelet base function for the medium-frequency gait component data to obtain optimal wavelet base parameters; and performing multi-layer wavelet packet decomposition on the medium-frequency gait component data based on the optimal wavelet base parameters to obtain a medium-frequency wavelet packet coefficient matrix;

[0047] Step S176: performing target frequency band baseline calibration on the medium-frequency gait component data to obtain target motion frequency band baseline parameters;

[0048] Step S177: performing target motion frequency band extraction and reconstruction according to the medium-frequency wavelet packet coefficient matrix and the target motion frequency band baseline parameters to obtain low-frequency target motion component data.

[0049] In this embodiment, the Open CV library is used to identify the motion source in the image sequence multi-band component data. Specifically, several common motion pattern templates are defined, such as image movement templates caused by operator walking and templates for autonomous motion of packages. The matchTemplate function of Open CV is used to match these templates with the image sequence multi-band component data frame by frame, thereby obtaining image motion source identification data. For example, by setting the matching threshold to 0.8, when the matching degree exceeds this threshold, it is considered that the corresponding motion source has been found. According to the image motion source identification data, the image sequence multi-band component data can be decoupled and separated using the NumPy library in Python. Specifically, according to the motion source identification data, the high-frequency jitter component in the image sequence multi-band component data is filtered out. For example, the frequency band caused by high-frequency jitter of the camera (assuming it is above 10 Hz) is separated to obtain high-frequency jitter component data. The jitter intensity of the high-frequency jitter component data is evaluated using functions in the signal analysis toolbox of MATLAB. For example, the root mean square (RMS) value of the high-frequency jitter component data is calculated as the high-frequency jitter intensity evaluation parameter. If the RMS value exceeds the preset threshold (assuming it is 0.5 pixels), it is considered that the jitter intensity is high. According to the evaluation results, a band-stop filter is designed using the FDA Toolbox of MATLAB, with the center frequency set to the jitter frequency and the bandwidth set to 2 Hz, thereby obtaining the filter configuration parameters of the jitter data. Based on the filter configuration parameters of the jitter data, the high-frequency jitter component data is band-stop filtered using the filter function in MATLAB. For example, the high-frequency jitter component data is processed using the designed band-stop filter, and the filtered data is subtracted from the original image sequence multi-band component data to obtain the medium-frequency gait component data. For the medium-frequency gait component data, the PyWavelets library is used to select the wavelet basis function and perform wavelet packet decomposition. Several common wavelet basis functions (such as db4, sym5, coif3) are tried, and the energy concentration degree of the medium-frequency gait component data after three-layer wavelet packet decomposition is calculated for each wavelet basis function. The wavelet basis function with the highest energy concentration degree is selected as the optimal wavelet basis parameter. For example, if the energy concentration degree of the db4 wavelet basis function after three-layer decomposition is the highest, then db4 is selected as the optimal wavelet basis parameter. Based on the optimal wavelet basis parameter, the medium-frequency gait component data is subjected to multi-layer wavelet packet decomposition to obtain the medium-frequency wavelet packet coefficient matrix. The target frequency band baseline of the medium-frequency gait component data is calibrated using the signal processing toolbox in MATLAB. According to the walking gait frequency range of the operator (assuming it is 1-3 Hz), the target motion frequency band is determined. The frequency spectrum of the medium-frequency gait component data is calculated using the frequency spectrum analysis function in MATLAB, and the energy peak point in the frequency band is found to determine the target motion frequency band baseline parameter.For example, if the energy peak point corresponds to a frequency of 2 Hz, the frequency and a certain range (such as ±0.5 Hz) near the frequency are taken as the target motion frequency band baseline parameters. According to the medium frequency wavelet packet coefficient matrix and the target motion frequency band baseline parameters, the target motion frequency band is extracted and reconstructed by using the PyWavelets library. Specifically, the medium frequency wavelet packet coefficient matrix is processed, and only the wavelet packet coefficients corresponding to the target motion frequency band are retained, and the coefficients of other frequency bands are discarded. By using the reconstruction function in the PyWavelets library, the processed wavelet packet coefficient matrix is inversely wavelet packet transformed to obtain low-frequency target motion component data.

[0050] Preferably, step S2 comprises the following steps:

[0051] Step S21: continuously collect three-axis acceleration and three-axis angular velocity data by a six-axis IMU sensor worn on the head of the operator at a sampling frequency of 500 Hz, with a range setting of ±16g and ±2000° / s, to obtain operator IMU inertial data;

[0052] Step S22: temperature drift compensation is performed on the operator IMU inertial data to obtain corrected operator IMU inertial data;

[0053] Step S23: complementary filtering fusion is performed on the corrected operator IMU inertial data to obtain operator head attitude angle data;

[0054] Step S24: head motion frequency spectrum transformation is performed on the operator based on the operator head attitude angle data to obtain head motion frequency spectrum feature data;

[0055] Step S25: gait cycle automatic recognition is performed on the operator according to the head motion frequency spectrum feature data to obtain operator gait phase segmentation points;

[0056] Step S26: periodic feature extraction is performed on the corrected operator IMU inertial data based on the operator gait phase segmentation points to obtain gait phase feature data;

[0057] Step S27: an operator plantar pressure signal is obtained; and variable gain image stabilization compensation is performed on the camera original image sequence according to the gait phase feature data and the operator plantar pressure signal to obtain a stabilized image sequence.

[0058] In this embodiment, in the dynamic sorting operation scenario of the logistics warehouse center, the MPU-9250 six-axis IMU sensor is equipped on the smart glasses worn by the operator. The sensor is connected to the main processor of the smart glasses through the I2C interface, continuously collecting three-axis acceleration and three-axis angular velocity data at a sampling frequency of 500 Hz. Specifically, the range of the accelerometer is set to ±16g, which can capture the large linear acceleration generated by the operator's head during rapid movement or sudden stop; the range of the gyroscope is set to ±2000° / s, which is used to accurately measure the rotational angular velocity of the head. These raw data are transmitted to the processor of the smart glasses through the I2C communication protocol to form the IMU inertial data set of the operator. For example, when the operator quickly turns around to check the packages on the other side of the shelf, the accelerometer will record the acceleration change of the head in the horizontal direction, and the gyroscope will capture the angular velocity information of the head rotation. To address the drift problem of the IMU sensor in different temperature environments, the "imufilter" function in MATLAB is used for temperature drift compensation. Specifically, the sensor is calibrated under different temperature conditions, and the relationship curve between temperature and drift is recorded. During actual data collection, the current temperature value is obtained in real time through the temperature meter of the sensor, and the acceleration and angular velocity data are dynamically corrected according to the preset temperature compensation coefficient. For example, when the temperature changes by 1℃, the accelerometer zero offset compensation value is set to 0.001g, and the gyroscope zero offset compensation value is set to 0.1° / s. The "pycomplmpl" filter library is used to fuse the acceleration and angular velocity data to obtain more accurate head posture angles. Specifically, the cutoff frequency of the low-pass filter is set to 0.1 Hz, so that the low-frequency components (such as gravity acceleration) in the acceleration data can pass smoothly; at the same time, the cutoff frequency of the high-pass filter is set to 5 Hz, so that the high-frequency components (such as rapid head movement) in the angular velocity data can pass. The filtered acceleration and angular velocity data are fused and calculated according to the complementary filtering formula to obtain the head posture angle data. For example, when the operator looks up to check the high shelf, the fused posture angle data can reflect the angle change of the head in the pitch direction in real time. Based on the head posture angle data, the "fft" function in the signal processing toolbox of MATLAB is used for head motion frequency spectrum transformation. The posture angle data is arranged in time sequence to form a time domain signal; the signal is processed with a Hanning window to reduce frequency spectrum leakage. The "fft" function is called to perform fast Fourier transform on the processed signal to obtain the frequency spectrum characteristic data of the head motion. For example, in the frequency spectrum graph, it can be clearly seen that the low-frequency motion components caused by the operator's walking gait are mainly concentrated in the range of 1-3 Hz, and the high-frequency head shaking components are distributed in the range of 5-15 Hz. These frequency spectrum characteristic data reflect the main frequency components of the head motion. According to the head motion frequency spectrum characteristic data, the "findpeaks" function in MATLAB is used to automatically identify the gait period.In actual operation, the spectral data is smoothed to remove noise interference; the minimum height of peak detection is set to 0.2 (relative to the maximum value of the spectral amplitude), to ensure that only significant peaks are identified; at the same time, the minimum interval is set to 50 sampling points (corresponding to 0.1 seconds), to avoid multiple peaks in the same cycle. Through these parameter settings, the "findpeaks" function can accurately identify the main peak positions in the spectrum, and then determine the phase segmentation points of the operator's gait. For example, in the spectral graph, the start and end points of each gait cycle correspond to specific peak positions. Based on the determined gait phase segmentation points, the IMU inertial data corrected in step S22 is processed again using the signal processing toolbox of MATLAB. Specifically, the IMU data is divided into multiple complete gait cycles according to the gait phase segmentation points. In each gait cycle, the mean, standard deviation of the acceleration data, and the maximum, minimum values of the angular velocity data, etc. are extracted. For example, the mean and standard deviation of the three-axis acceleration in each gait cycle are calculated to analyze the motion stability and amplitude variation of the operator's head in different directions; at the same time, the peak values of the angular velocity data are counted to understand the severity of head rotation. Through statistical analysis of these characteristics, gait phase feature data is obtained. The detailed implementation process of step S27 can be referred to the sub-steps of step S27.

[0059] Preferably, step S27 comprises the following steps:

[0060] Step S271: Obtain the operator's plantar pressure signal; calculate the pressure center trajectory of the operator's plantar pressure signal to obtain plantar pressure gravity data;

[0061] Step S272: Time sequence correlation modeling is performed according to the gait phase feature data and the plantar pressure gravity data to obtain an operator gait pressure correlation model;

[0062] Step S273: Neck muscle tension prediction is performed based on the operator gait pressure correlation model to obtain operator neck tremor prediction parameters;

[0063] Step S274: A variable gain control strategy is constructed according to the operator neck tremor prediction parameters to obtain camera image stabilization compensation parameters;

[0064] Step S275: Mechanical image stabilization driving is performed on the intelligent camera based on the camera image stabilization compensation parameters to obtain mechanical compensation displacement data;

[0065] Step S276: Electronic image stabilization fusion processing is performed on the mechanical compensation displacement data to obtain camera fusion image stabilization parameters;

[0066] Step S277: Real-time image stabilization processing is performed on the camera original image sequence based on the camera fusion image stabilization parameters to obtain a stabilized image sequence.

[0067] In this embodiment, in the dynamic sorting operation of the logistics warehouse center, the operator wears a plantar pressure sensor (such as Novel Pedar-X system) to obtain plantar pressure signals. The sensor samples at a frequency of 100 Hz and sends data to the processor of the smart glasses through wireless transmission. In the processor, the plantar pressure signal is calculated by using the signal processing toolbox of MATLAB. Specifically, by calculating the plantar pressure distribution of each step, the position change trajectory of the pressure center is determined, and the plantar pressure barycenter data is obtained. For example, during the walking process of the operator, the pressure center moves back and forth on the plantar surface. By integrating and calculating the coordinates of the pressure distribution of each step, the trajectory data of the plantar pressure barycenter changing with time is obtained. According to the obtained gait phase feature data and the obtained plantar pressure barycenter data, a time series correlation modeling is performed by using the scikit-learn library in Python. Specifically, the gait phase feature data and the plantar pressure barycenter data are used as input features to build a time series model (such as LSTM neural network). In the model training process, the number of hidden layer units is set to 50, the learning rate is set to 0.001, and the training period is set to 100 times. The operator gait pressure correlation model obtained by training can predict the relationship between gait phase and plantar pressure. By using the established operator gait pressure correlation model, the neck muscle tension is predicted in combination with a biomechanical model (such as OpenSim). In OpenSim, the operator's neck muscle model parameters are set, including the maximum strength and contraction speed of the muscle. According to the gait phase and plantar pressure information output by the gait pressure correlation model, the stress of the neck muscle in different gait stages is simulated. For example, during the swing phase of the operator's walking, the model predicts that the tension of the neck muscle is low; while in the support phase, the tension of the neck muscle is high. Through simulation, the operator's neck tremor prediction parameters are obtained, including the average value and standard deviation of muscle tension. According to the obtained operator's neck tremor prediction parameters, a variable gain control strategy is constructed by using the control toolbox in MATLAB. Specifically, the parameters of the PID controller are set, in which the proportional gain is 1.2, the integral gain is 0.5, and the differential gain is 0.3. According to the neck tremor prediction parameters, the gain value of the controller is adjusted. When the neck tremor is predicted to be large, the gain is increased to enhance the image stabilization compensation effect; otherwise, the gain is reduced. Through the above steps, the camera image stabilization compensation parameters are obtained, including the compensation amplitude and compensation frequency. Based on the obtained camera image stabilization compensation parameters, mechanical compensation is performed by using the mechanical image stabilization driving system (such as a gimbal) of the smart camera. Specifically, the compensation parameters are sent to the control motor of the gimbal, and the precise driving of the motor is used to realize the mechanical displacement compensation of the camera. For example, the compensation parameters indicate that the gimbal compensates 0.5 degrees in the X-axis direction and 0.3 degrees in the Y-axis direction, and the gimbal motor rotates accordingly according to these instructions, thereby reducing the image jitter caused by the operator's neck tremor.The actual displacement data of the cloud head is recorded as mechanical compensation displacement data. The obtained mechanical compensation displacement data is subjected to electronic image stabilization fusion processing, and an image stabilization module in the Open CV library is adopted. Specifically, a transformation matrix between the image sequence after mechanical compensation and the original image sequence is calculated by using the "estimateRigidTransform" function of Open CV, and then an affine transformation is performed on the image by using the "warpAffine" function, so as to further eliminate residual jitter. At the same time, the parameters of electronic image stabilization, such as compensation gain and filtering strength, are adjusted in combination with the mechanical compensation displacement data. Finally, the fused camera image stabilization parameters, including the comprehensive compensation amplitude and the compensation angle, are obtained. Based on the obtained camera fusion image stabilization parameters, the "Stitcher" class of Open CV is used to perform real-time image stabilization processing on the original image sequence of the camera. Specifically, the fusion image stabilization parameters are applied to each frame of image, and through image registration and fusion algorithm, the image sequence after image stabilization compensation is generated. For example, the compensation parameters are applied to each frame of image, and the position and angle of the image are adjusted, so that the finally output image sequence is more visually stable, and the jitter caused by the shaking of the operator's head and neck is reduced.

[0068] Preferably, step S3 comprises the following steps:

[0069] Step S31: Collecting multi-dimensional perception data of the operator's surrounding environment at a synchronous sampling frequency of 500 Hz by integrating the sensor array of the IMU gyroscope, the three-axis magnetometer, the ultrasonic ranging sensor and the laser radar, to generate multi-modal operation environment data;

[0070] Step S32: Aligning and calibrating the multi-modal operation environment data to obtain synchronous multi-modal operation environment data;

[0071] Step S33: Identifying noise feature parameters of the sensor based on the synchronous multi-modal operation environment data;

[0072] Step S34: Filtering the synchronous multi-modal operation environment data based on the sensor noise feature parameters to obtain denoised multi-modal operation environment data;

[0073] Step S35: Constructing an operator multi-modal state vector based on the denoised multi-modal operation environment data;

[0074] Step S36: Designing Kalman filter parameters based on the operator multi-modal state vector to obtain Kalman filter configuration parameters; and performing Kalman filtering on the denoised multi-modal operation environment data according to the Kalman filter configuration parameters to obtain operator motion state data;

[0075] Step S37: target object optical flow calculation is performed on the stabilized compensation image sequence according to the operator motion state data, to obtain a target motion vector field.

[0076] In this embodiment, in the logistics warehouse center, the smart glasses worn by the operator are equipped with various sensors, including MPU-6000 IMU gyroscope, HMC5883L three-axis magnetometer, HC-SR04 ultrasonic ranging sensor, and VL53L0X laser radar. These sensors work at a synchronous sampling frequency of 500 Hz, and the data is synchronously collected by the main processor of the smart glasses to generate multi-modal operating environment data. For example, the IMU gyroscope provides the angular velocity information of the head, the magnetometer provides the magnetic field direction information, the ultrasonic sensor measures the distance from the surrounding objects, and the laser radar is used for accurate ranging. The multi-modal operating environment data is aligned and calibrated to obtain synchronous multi-modal operating environment data. In actual operation, the NumPy library and SciPy library in Python are used for data alignment and calibration. Specifically, the timestamps of each sensor are determined, and then all sensor data is adjusted to a unified time axis through interpolation methods. For example, the data of the ultrasonic ranging sensor and the laser radar is aligned in time with the data of the IMU gyroscope and the magnetometer through linear interpolation. In addition, the known sensor position and direction relationship is used to correct the data geometrically to ensure the consistency of the data of different sensors in space. The noise feature parameters of the sensors are obtained by identifying the noise features of the synchronous multi-modal operating environment data. In this step, the "signalAnalyzer" application in the signal analysis toolbox of MATLAB can be used to identify noise features. For example, by analyzing the frequency spectrum of the IMU gyroscope data, it is determined that the noise is mainly concentrated in the high frequency band, showing random high frequency fluctuations. For the magnetometer data, the noise includes periodic fluctuations caused by environmental electromagnetic interference. The data of the ultrasonic ranging sensor is affected by the multipath effect caused by environmental reflection, showing random jumps in the measured values. By statistical analysis of these noise features, the noise mean, variance, and noise power spectral density of each sensor are calculated. Based on the sensor noise feature parameters, the synchronous multi-modal operating environment data is filtered to obtain denoised multi-modal operating environment data. In this step, suitable filtering methods are adopted according to the data characteristics of different sensors. For example, for the IMU gyroscope data, a low-pass filter is used to remove high-frequency noise, with a cutoff frequency of 30 Hz to retain valid motion information. For the magnetometer data, a band-stop filter is used to suppress electromagnetic interference at a specific frequency, such as a band-stop filter at 50 Hz or 60 Hz to remove power frequency interference. For the ultrasonic ranging sensor data, a median filter is used to remove random jump noise points with a window size of 5 data points. The NumPy library in Python is used to construct the state vector of the denoised multi-modal operating environment data. Specifically, the three-axis angular velocity of the IMU gyroscope, the three-axis magnetic field strength of the magnetometer, the distance measurement value of the ultrasonic ranging sensor, and the ranging data of the laser radar are integrated into a state vector.For example, the state vector can be represented as [x_acc, y_acc, z_acc, x_gyro, y_gyro, z_gyro, x_mag, y_mag, z_mag, ultrasound_dist, lidar_dist], where each element represents the corresponding sensor measurement. The denoised multi-modal operating environment data is Kalman filtered according to the Kalman filter configuration parameters to obtain operator motion state data. In this step, the “designKalmanFilter” function of MATLAB is used to design the Kalman filter. Specifically, according to the dimension of the state vector and the dynamic characteristics of the system, the state transition matrix, the observation matrix, the process noise covariance matrix and the measurement noise covariance matrix are determined. For example, for a simple linear system, the state transition matrix can be set as a unit matrix plus a small increment matrix related to the time step. The measurement noise covariance matrix is set according to the previously identified sensor noise characteristic parameters, such as the noise variance of the IMU gyroscope being 0.01 (° / s)^2, the noise variance of the magnetometer being 0.1 μT^2, and the noise variances of the ultrasonic ranging sensor and the laser radar being 0.5 cm^2 and 0.1 cm^2 respectively. For details of the implementation process of step S37, please refer to the sub-steps of step S37.

[0077] Preferably, step S37 comprises the following steps:

[0078] Step S371: motion pattern feature extraction is performed on the operator motion state data to obtain an operator motion feature vector; motion pattern recognition modeling is performed based on the operator motion feature vector to obtain an operator motion state classification model;

[0079] Step S372: real-time classification is performed on the operator motion state data according to the operator motion state classification model to obtain an operator motion state recognition result;

[0080] Step S373: state transition statistics are performed on the operator motion state recognition result to obtain an operator motion state transition probability table;

[0081] Step S374: smoothing is performed on the operator motion state recognition result based on the operator motion state transition probability table to obtain operator stable motion state data;

[0082] Step S375: visual algorithm parameter mapping is performed according to the operator stable motion state data to obtain a visual algorithm parameter configuration mapping table;

[0083] Step S376: parameter configuration adjustment is performed on the image sequence based on the visual algorithm parameter configuration mapping table to obtain image sequence visual parameter configuration data;

[0084] Step S377: Pyramid optical flow calculation is performed on the stabilized compensation image sequence according to the image sequence visual parameter configuration data, to obtain a target motion vector field.

[0085] In this embodiment, in the dynamic sorting operation of the logistics warehouse center, the motion state data collected by the smart glasses worn by the operator includes position, speed and attitude information. The scikit-learn library in Python is used to extract motion pattern features from these data. Specifically, by calculating the mean, standard deviation of acceleration and the peak value of angular velocity, an operator motion feature vector is constructed. For example, the motion feature vector can be represented as [mean_acceleration, std_acceleration, max_angular_velocity]. Based on these feature vectors, a support vector machine (SVM) is used for motion pattern recognition modeling. In the SVM model, the radial basis function (RBF) is set as the kernel function, the kernel function parameter γ is set to 0.1, and the regularization parameter C is set to 10. By training the SVM model, an operator motion state classification model is obtained, which can classify the operator's motion state into categories such as walking, running, and stationary. According to the operator motion state classification model, the scikit-learn library in Python is used to classify the real-time collected operator motion state data. Specifically, the motion feature vector extracted from the real-time data is input into the trained SVM model, and the model outputs the current motion state category of the operator. For example, when the operator is walking between shelves, the model outputs the "walking" state; when the operator stops moving, the model outputs the "stationary" state. By continuously classifying real-time data, the operator motion state recognition result is obtained. The operator motion state recognition result is used for state transition statistics, and the Pandas library in Python is used to construct an operator motion state transition probability table. Specifically, the number of times each motion state transitions to other states is counted, and then the transition probability is calculated. For example, in 100 state changes, from the "walking" state to the "stationary" state, there are 30 times, to the "running" state, there are 20 times, and to maintain the "walking" state, there are 50 times. According to these statistical data, a state transition probability table is constructed, which records the probability of transitioning from each state to other states. Based on the operator motion state transition probability table, the Hidden Markov Model (HMM) library (such as hmmlearn) in Python is used to smooth the operator motion state recognition result. Specifically, the state transition probability table is used as the transition matrix of HMM, the initial state probability is set, and the latent state sequence is calculated according to the real-time motion state recognition result. For example, if the "walking", "stationary", "walking" states appear continuously, the HMM model will judge that the "stationary" state in the middle is noise according to the transition probability table, and thus it is smoothed and corrected to the "walking" state. According to the stable motion state data of the operator, the NumPy library in Python is used for visual algorithm parameter mapping to obtain a visual algorithm parameter configuration mapping table.Specifically, corresponding visual algorithm parameters are set for each motion state. For example, in the "walking" state, the pyramid layer number of the optical flow method is set to 3, and the window size is set to 15x15; in the "running" state, the pyramid layer number is set to 4, and the window size is set to 20x20. These parameter settings are based on the motion speed and stability requirements of the target object under different motion states. Through such a mapping relationship, a visual algorithm parameter configuration mapping table is constructed. Based on the visual algorithm parameter configuration mapping table, the Open CV library in Python is used to adjust the parameters of the image stabilization compensation image sequence. Specifically, the corresponding visual algorithm parameters such as the pyramid layer number and the window size are obtained from the mapping table according to the current motion state, and these parameters are applied to the optical flow calculation. For example, when the operator is in the "walking" state, the pyramid layer number 3 and the window size 15x15 are used for optical flow calculation to obtain image sequence visual parameter configuration data. According to the image sequence visual parameter configuration data, the sparse optical flow function "calcOpticalFlowPyrLK" in the Open CV library is used to calculate the pyramid optical flow. Specifically, the image stabilization compensation image sequence is used as input, combined with the configured pyramid layer number and window size, to calculate the motion vector field of the target object in the image sequence. For example, in the "walking" state, the set pyramid layer number 3 and window size 15x15 are used to calculate the displacement vector of each feature point between consecutive frames to obtain the target motion vector field. This vector field describes the motion direction and speed of the target object in detail.

[0086] Preferably, step S4 comprises the following steps:

[0087] Step S41: Perform vector field density calculation on the target motion vector field to obtain a motion vector density distribution map;

[0088] Step S42: Perform preliminary screening of the target candidate region based on the motion vector density distribution map to obtain candidate target motion region data;

[0089] Step S43: Perform connectivity detection on the candidate target motion region data to obtain a connected target motion region label;

[0090] Step S44: Perform time sequence correlation on the target motion vector field based on the connected target motion region label to obtain an initial target trajectory candidate sequence;

[0091] Step S45: Perform stability screening of the initial target trajectory candidate sequence according to the low-frequency target motion component data to obtain a target trajectory candidate sequence;

[0092] Step S46: trajectory smoothness evaluation is performed on the target trajectory candidate sequence to obtain a trajectory smoothness evaluation index; quality screening is performed on the target trajectory candidate sequence according to the trajectory smoothness evaluation index to obtain a target trajectory candidate point set;

[0093] Step S47: RANSAC outlier elimination is performed on the target trajectory candidate point set to obtain an effective target trajectory; physical constraint verification is performed on the effective target trajectory to obtain a target real motion trajectory.

[0094] In this embodiment, in the dynamic sorting operation of the logistics warehouse center, the target motion vector field data collected by the smart glasses contains a large amount of motion information of the target objects. The NumPy library in Python is used for vector field density calculation. Specifically, the image is divided into multiple small grid regions, for example, each grid region is 10x10 pixels. In each grid, the number of target motion vectors falling into the region is counted, and the number of vectors per unit area, i.e. the vector density, is calculated. For example, if there are 8 vectors in a grid region, the vector density of the region is 0.08 vectors / pixel2. By traversing the entire image, a motion vector density distribution map is generated, which reflects the motion activity level of different regions in the image. Based on the motion vector density distribution map, the threshold segmentation function in the Open CV library is used for preliminary screening of the target candidate regions. Specifically, a density threshold is set, for example 0.05 vectors / pixel2. Regions with a vector density higher than the threshold are marked as target candidate regions. For example, in the image, regions with a vector density higher than 0.05 correspond to packages being moved quickly by the operator. By traversing the entire vector density distribution map, all regions that meet the conditions are screened out to obtain candidate target motion region data. Connectivity detection is performed on the candidate target motion region data using the connected region labeling function in the Open CV library. Specifically, a connectivity judgment threshold is set, for example 8 connectivity criteria. Traverse the candidate target motion region data and mark adjacent and density continuous regions as the same connected target motion region. For example, in the image, if two high-density regions are adjacent under the 8 connectivity criteria, they will be marked as the same connected region. Through the above steps, the connected target motion region label is obtained. Based on the connected target motion region label, the Pandas library in Python is used to perform time series association on the target motion vector field. Specifically, each connected region is assigned a unique identifier and its occurrence in the time series is recorded. For example, if a connected region appears in consecutive 3 frames of images, it is recorded as an initial target trajectory candidate sequence. Through the above steps, the connected regions in space are extended to the time dimension to construct the initial target trajectory candidate sequence. According to the low-frequency target motion component data, the SciPy library in Python is used to perform stability screening on the initial target trajectory candidate sequence. Specifically, the correlation coefficient of each trajectory candidate sequence with the low-frequency target motion component data is calculated. For example, set the correlation coefficient threshold to 0.7. If the correlation coefficient of the trajectory candidate sequence with the low-frequency motion component is higher than the threshold, it is considered that the trajectory has higher stability. Through the above steps, the target trajectory candidate sequence consistent with the low-frequency motion is screened out. The trajectory smoothness of the target trajectory candidate sequence is evaluated using the curve fitting function in the Open CV library. Specifically, a polynomial fitting is performed on each trajectory candidate sequence, for example, a quadratic polynomial fitting.The mean square error (MSE) of the fitting curve is calculated as an evaluation index of the trajectory smoothness. For example, the threshold of the MSE is set as 0.5 pixel2. If the MSE of the trajectory is lower than the threshold, the trajectory is considered to have good smoothness. Through the above steps, the target trajectory candidate point set is obtained by quality screening of the target trajectory candidate sequence according to the trajectory smoothness evaluation index. For details of the implementation process of step S47, refer to the sub-steps of step S47.

[0095] Especially important is that step S47 further includes the following steps:

[0096] Step S471: RANSAC anomaly detection initialization is performed on the target trajectory candidate point set to obtain RANSAC algorithm initialization parameters; and random sampling modeling is performed on the target trajectory candidate point set based on the RANSAC algorithm initialization parameters to obtain a target motion model hypothesis set.

[0097] Step S472: consistency verification is performed on the target trajectory candidate point set according to the target motion model hypothesis set to obtain target trajectory inlier marking data.

[0098] Step S473: optimal model selection is performed on the target motion model hypothesis set based on the target trajectory inlier marking data to obtain an optimal target motion trajectory model.

[0099] Step S474: anomaly point elimination is performed on the target trajectory candidate point set according to the optimal target motion trajectory model to obtain effective target trajectory data.

[0100] Step S475: physical constraint condition verification is performed on the effective target trajectory data to obtain a physical rationality evaluation result; and constraint screening is performed on the effective target trajectory data based on the physical rationality evaluation result to obtain constraint-compliant trajectory data.

[0101] Step S476: resonance feature detection is performed on the constraint-compliant trajectory data to obtain a resonance feature detection result.

[0102] Step S477: pseudo motion marking is performed on the constraint-compliant trajectory data according to the resonance feature detection result to obtain target real motion trajectory data.

[0103] In this embodiment, the RANSAC (Random Sample Consensus) algorithm in the statistical and machine learning toolbox of MATLAB is used for anomaly detection of the target trajectory candidate point set. The RANSAC anomaly detection initialization is performed on the target trajectory candidate point set, and the initial parameters are set: the sampling number is 100 times, the threshold distance is 2 pixels, and the confidence is 0.99. Based on these initialization parameters, random sampling modeling is performed on the target trajectory candidate point set to obtain multiple target motion model hypothesis sets. For example, in each sampling, enough points are randomly selected to fit a motion model, and after 100 samplings, multiple different motion model hypotheses are obtained. The target trajectory candidate point set contains a large number of trajectory points, which are affected by noise or abnormal factors. Specifically, the target trajectory candidate point set is preprocessed to remove obviously deviated points. The parameters of the RANSAC algorithm are set, including the sampling number (100 times), the threshold distance (2 pixels), and the confidence (0.99). In each sampling, enough points are randomly selected to fit a motion model, such as a linear model or a polynomial model. After 100 samplings, multiple different motion model hypothesis sets are obtained, each of which represents a motion trajectory. According to the target motion model hypothesis set, the consistency of the target trajectory candidate point set is verified using the Open CV library in Python. Specifically, each model hypothesis is applied to all trajectory points, and the distance between each point and the predicted position of the model is calculated. If the distance is less than the set threshold (e.g., 2 pixels), the point is considered consistent with the model and is marked as an inlier. Through the above steps, the target trajectory inlier marking data is obtained. The target motion model hypothesis set contains multiple motion models, which need to be verified. Using the Open CV library in Python, each model hypothesis is applied to all trajectory points. For each trajectory point, the distance between it and the predicted position of the model is calculated. If the distance is less than the set threshold (e.g., 2 pixels), the point is considered consistent with the model and is marked as an inlier; otherwise, it is marked as an outlier. Through the above steps, the inlier and outlier data corresponding to each model are obtained. The inlier data will be used for subsequent model selection, while the outlier data is considered as an abnormal point. Based on the target trajectory inlier marking data, the Open CV library model evaluation function is used again to select the optimal model from the target motion model hypothesis set. Specifically, the number of inliers corresponding to each model is counted, and the model with the most inliers is selected as the optimal target motion trajectory model. For example, if there are three model hypotheses, the corresponding inlier numbers are 80, 120, and 95, respectively, and the model with the inlier number of 120 is selected as the optimal model. Each model in the target motion model hypothesis set has corresponding inlier and outlier data. In order to select the optimal model, the Open CV library model evaluation function is used. Specifically, the number of inliers corresponding to each model is counted.The model with the most inliers usually represents the trajectory that best fits the real motion pattern. For example, if there are three models with corresponding inlier counts of 80, 120, and 95, the model with 120 inliers is selected as the optimal target motion trajectory model. Based on the optimal target motion trajectory model, the NumPy library is used to remove outliers from the target trajectory candidate point set. Specifically, all trajectory points are compared with the optimal model, and points that are more than a threshold (e.g., 3 pixels) away from the model's predicted position are removed. The optimal target motion trajectory model represents the trajectory that best fits the real motion pattern. The NumPy library is used to remove outliers from the target trajectory candidate point set. Specifically, all trajectory points are compared with the optimal model, and the distance of each point to the model's predicted position is calculated. If the distance exceeds a set threshold (e.g., 3 pixels), the point is considered an outlier and is removed. Through the above steps, the effective target trajectory data is obtained. The physical constraint module in Python (such as PyBullet) is used to test the physical constraint conditions of the effective target trajectory data. Specifically, the trajectory data is tested for compliance with Newton's laws of motion, such as whether the acceleration is within a reasonable range (e.g., the maximum acceleration does not exceed 5 m / s 2 ). Based on the test results, the physical reasonableness evaluation results are obtained, and based on these results, the effective target trajectory data is further filtered to obtain constraint-compliant trajectory data, i.e., trajectory points that comply with physical laws. Although the effective target trajectory data is highly consistent with the optimal model, it still needs to be tested for compliance with physical laws. The physical constraint module in Python (such as PyBullet) is used to test the physical reasonableness of the trajectory data. Specifically, the acceleration in the trajectory data is calculated, and it is checked whether it is within a reasonable range (e.g., the maximum acceleration does not exceed 5 m / s 2If the acceleration exceeds a reasonable range, the trajectory point is considered to be inconsistent with physical laws and is removed. Through the above steps, the constraint-compliant trajectory data, i.e., those trajectory points that comply with physical laws, is obtained. The constraint-compliant trajectory data is subjected to resonance feature detection using the resonance detection function in the signal processing toolbox of MATLAB. Specifically, the trajectory data is subjected to spectral analysis to identify whether there is a resonance frequency (e.g., whether there is a resonance component close to the operator's gait frequency). The resonance feature detection result is obtained, which helps to identify and exclude pseudo-motion trajectories caused by system resonance. The constraint-compliant trajectory data, although complying with physical laws, still has pseudo-motion trajectories caused by system resonance. In order to identify these pseudo-motion trajectories, the resonance detection function in the signal processing toolbox of MATLAB is used. Specifically, the trajectory data is subjected to spectral analysis to identify whether there is a resonance frequency (e.g., whether there is a resonance component close to the operator's gait frequency). If a resonance frequency is detected, the trajectory data is considered to be affected by pseudo-motion. Through the above steps, the resonance feature detection result is obtained. According to the resonance feature detection result, the constraint-compliant trajectory data is labeled for pseudo-motion using a data labeling library (such as Pandas) in Python. Specifically, the trajectory data segments that are detected to have resonance features are labeled to indicate that these data segments are affected by pseudo-motion. Finally, the data segments that are not labeled are the target real motion trajectory data, which comply with physical laws and are not affected by resonance, accurately reflecting the real motion of the target object. The resonance feature detection result indicates that certain trajectory data segments are affected by pseudo-motion. In order to further process these data, the constraint-compliant trajectory data is labeled for pseudo-motion using a data labeling library (such as Pandas) in Python. Specifically, the trajectory data segments that are detected to have resonance features are labeled to indicate that these data segments are affected by pseudo-motion. Finally, the data segments that are not labeled are the target real motion trajectory data, which comply with physical laws and are not affected by resonance, accurately reflecting the real motion of the target object.

[0104] Preferably, step S5 comprises the following steps:

[0105] Step S51: target region bounding box positioning based on the target real motion trajectory, to obtain target bounding box coordinate data;

[0106] Step S52: target region cropping of the stabilized compensation image sequence according to the target bounding box coordinate data, to obtain a target image block sequence;

[0107] Step S53: color histogram feature extraction of the target image block sequence, to obtain a target color feature vector;

[0108] Step S54: HOG gradient direction feature calculation is performed based on the target image block sequence to obtain a target HOG feature vector;

[0109] Step S55: LBP texture pattern recognition is performed on the target image block sequence to obtain a target LBP feature vector;

[0110] Step S56: The target comprehensive feature vector is obtained by splicing the target color feature vector, the target HOG feature vector and the target LBP feature vector;

[0111] Step S57: Multi-target fusion tracking is performed on the target object according to the target comprehensive feature vector to obtain a target positioning recognition result.

[0112] In this embodiment, the "cv2.boundingRect" function in the Open CV library can be used to locate the target region bounding box based on the target real motion trajectory. The coordinate point set of the target real motion trajectory is input, and the function calculates the smallest bounding box that completely contains all the trajectory points to obtain the target bounding box coordinate data (x, y, width, height), where (x, y) is the coordinate of the upper left corner of the bounding box, and width and height are the width and height of the bounding box, respectively. For example, if the trajectory point set is [[100, 200], [150, 250], [200, 200]], the obtained bounding box coordinate data is x = 100, y = 200, width = 100, and height = 50. According to the target bounding box coordinate data, the "cv2.getRectSubPix" function in the Open CV library is used to crop the target region from the stabilized compensation image sequence. The stabilized compensation image and the bounding box coordinate data are input, and the function crops the region specified by the bounding box from the image to obtain the target image block sequence. For example, if the size of the stabilized compensation image is 640x480 pixels and the bounding box coordinate data is x = 100, y = 200, width = 100, and height = 50, the size of the cropped target image block is 100x50 pixels. The "cv2.calcHist" function in the Open CV library can be used to extract color histogram features from the target image block sequence. Each image in the target image block sequence is input, and parameters such as color channel, mask, histogram dimension, and value range are set to calculate the color histogram of each image and obtain the target color feature vector. For example, the histogram of each image is calculated for the RGB three channels, the histogram dimension of each channel is 256, and the value range is [0, 255]. The histograms of the three channels are concatenated into a 768-dimensional vector as the target color feature vector. The "cv2.HOGDescriptor" class in the Open CV library can be used to calculate the HOG gradient direction features based on the target image block sequence. The HOG descriptor object is created, and parameters such as cell size, block size, step size, and gradient direction number are set. For example, the cell size is set to 8x8 pixels, the block size is set to 16x16 pixels, the step size is set to 8x8 pixels, and the gradient direction number is set to 9. The HOG feature of each image in the target image block sequence is calculated to obtain the target HOG feature vector. The "cv2.texture" module in the Open CV library or a custom LBP calculation function can be used for LBP texture pattern recognition on the target image block sequence. The neighborhood radius and the number of sampling points of LBP are set, for example, the neighborhood radius is 3 pixels and the number of sampling points is 8. LBP is calculated for each image to obtain the LBP value of each pixel. Then the LBP histogram is calculated, and the normalized histogram is used as the target LBP feature vector.For example, the dimension of the LBP histogram is 256, which is normalized into a 256-dimensional vector representing the frequency of different LBP patterns. According to the target color feature vector, the target HOG feature vector, and the target LBP feature vector, the feature vectors are spliced. The "np.hstack" function or "np.concatenate" function in the NumPy library can be used. The three feature vectors are spliced as input according to the dimension order to obtain the target comprehensive feature vector. For example, the color feature vector has a dimension of 768, the HOG feature vector has a dimension of 909, and the LBP feature vector has a dimension of 256. The spliced comprehensive feature vector has a dimension of 768+909+256=1933. For details of the implementation process of step S57, please refer to the sub-steps of step S57.

[0113] Especially important is that step S57 further comprises the following steps:

[0114] Step S571: Image quality evaluation is performed on the stabilized compensation image sequence to obtain an image quality score set; and a weight distribution strategy is constructed based on the image quality score set to obtain a target feature weight distribution rule;

[0115] Step S572: The target comprehensive feature vector is weighted processed according to the target feature weight distribution rule to obtain current frame fusion feature weight data;

[0116] Step S573: The target feature template is constructed based on the current frame fusion feature weight data to obtain target template feature data;

[0117] Step S574: Similarity calculation is performed on the target template feature data and the current frame fusion feature weight data to obtain target feature matching similarity;

[0118] Step S575: The target position is predicted according to the target feature matching similarity to obtain target position prediction data; and the prediction region expansion search is performed based on the target position prediction data to obtain search region candidate position data;

[0119] Step S576: Multi-feature matching verification is performed on the search region candidate position data to obtain best matching position data;

[0120] Step S577: The tracking result fusion is performed according to the best matching position data and the target real motion trajectory data to obtain the camera target positioning recognition result.

[0121] In this embodiment, image quality assessment is performed on a sequence of image stabilization-compensated images. The "cv2.Laplacian" function in the Open CV library can be used to calculate the image's Laplacian operator, and the image clarity is statistically analyzed to obtain an image quality score set. Specifically, the Laplacian operator is applied to each image, and its variance is calculated. A larger variance indicates a clearer image. Based on the resulting image quality score set, a weight allocation strategy is constructed. For example, the scores are normalized using the "np.normalize" function in the NumPy library to obtain a target feature weight allocation rule. For example, if the image quality score set is [100, 200, 150], the weight allocation rule obtained after normalization is [0.2, 0.4, 0.3], indicating that the target feature weights for the corresponding images are 20%, 40%, and 30%, respectively. Based on the target feature weight allocation rule, the "np.multiply" function in the NumPy library is used to weight the target comprehensive feature vector. The target comprehensive feature vector is multiplied by the corresponding weight vector to obtain the current frame fusion feature weight data. For example, if the target's comprehensive feature vector is [0.1, 0.2, 0.3] and the weight assignment rule is [0.2, 0.4, 0.3], then the weighted fusion feature weights for the current frame are [0.02, 0.08, 0.09]. Based on the fusion feature weights for the current frame, the "cv2.ml" module in the Open CV library is used to construct the target feature template. Specifically, the fusion feature weights for multiple frames are averaged to generate the target template feature data. For example, the fusion feature weights for the past 10 frames are collected and averaged to generate the target template feature data. This template feature data serves as the standard feature for the target. The "scipy.spatial.distance.cosine" function in the SciPy library is used to calculate the similarity between the target template feature data and the fusion feature weights for the current frame. The target feature matching similarity is calculated by calculating the cosine distance between the two vectors. For example, if the target template feature data is [0.03, 0.06, 0.07] and the current frame's fused feature weights are [0.02, 0.08, 0.09], the similarity calculation result is 1-cosine([0.03, 0.06, 0.07], [0.02, 0.08, 0.09]), which yields the target feature matching similarity. Based on this target feature matching similarity, the "cv2.matchTemplate" function in the Open CV library is used to predict the target's position. Using this similarity as the basis for matching the template, the current frame is searched for the best matching area to obtain the target position prediction data. Based on this target position prediction data, the "np.expand_dims" function in the NumPy library is used to perform an expanded search of the prediction area, yielding candidate search area locations.For example, in the current frame, if the target position prediction data is (x = 100, y = 200), a certain range (such as 50 pixels) is expanded around the position to search and obtain search area candidate position data. For the search area candidate position data, a multi-feature matching verification is performed using the "cv2.HistogramBackProject" function in the Open CV library. Specifically, the matching degrees of color, HOG and LBP features in the candidate area are calculated respectively, and the matching results of the three features are integrated to obtain the best matching position data. For example, the matching degrees of the three features are calculated for each candidate position, the average value is taken as the comprehensive matching degree, and the position with the highest comprehensive matching degree is selected as the best matching position data. According to the best matching position data and the target real motion trajectory data, the tracking result fusion is performed using the "cv2Kalman" class in the Open CV library. Specifically, the best matching position data is input as the measurement value into the Kalman filter, and the target real motion trajectory data is combined as the prediction value to obtain the final camera target positioning recognition result. For example, the initial state of the Kalman filter is set as the current position and speed of the target real motion trajectory, the measurement noise covariance matrix and the process noise covariance matrix are adjusted according to experimental data, and the filter state is updated by iteration to obtain a smooth and accurate target positioning recognition result.

[0122] Preferably, the present application provides an intelligent camera positioning recognition system for performing the intelligent camera positioning recognition method as described above, which comprises:

[0123] An image preprocessing module is configured to acquire a camera original image sequence, perform multi-band decoupling on the camera original image sequence to obtain image sequence multi-band component data, and perform low-frequency target motion component recognition on the image sequence multi-band component data to obtain low-frequency target motion component data.

[0124] A steady image compensation module is configured to acquire operator IMU inertial data, perform gait cycle recognition on the operator IMU inertial data to obtain gait phase feature data, and perform variable gain steady image compensation on the camera original image sequence according to the gait phase feature data to obtain a steady image compensation image sequence.

[0125] A motion estimation module is configured to acquire multi-modal operating environment data, perform motion state estimation on an operator based on the multi-modal operating environment data to obtain operator motion state data, and perform target object optical flow calculation on the steady image compensation image sequence according to the operator motion state data to obtain a target motion vector field.

[0126] The trajectory evaluation and screening module is configured to perform motion trajectory evaluation on the target motion vector field to obtain a target trajectory candidate point set, perform RANSAC outlier elimination on the target trajectory candidate point set to obtain an effective target trajectory, and perform physical constraint verification on the effective target trajectory to obtain a target real motion trajectory.

[0127] The target recognition and tracking module is configured to perform target object multi-feature extraction based on the target real motion trajectory to obtain a target comprehensive feature vector, and perform multi-target fusion tracking on the target object based on the target comprehensive feature vector to obtain a target positioning recognition result.

[0128] Preferably, the present application further provides an intelligent glasses, the intelligent glasses comprising: a processor, a memory, a camera, a display screen, a communication circuit, an audio component and a ranging component; the camera comprising a visible light camera and an infrared camera; the memory stores a program capable of being loaded and executed by the processor to implement the intelligent camera positioning recognition method as described above.

[0129] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, the scope of the present application being defined by the appended claims and not by the above description, therefore all variations falling within the meaning and scope of the equivalent elements of the application file are intended to be included in the present application.

[0130] The above description is merely one specific implementation of the present application, enabling those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for positioning and identifying an intelligent camera, characterized in that: The following steps are involved: Step S1: Obtain the original image sequence of the camera; Perform multi-band decoupling on the original image sequence of the camera to obtain multi-band component data of the image sequence; Performing low-frequency target motion component recognition on multi-band component data of the image sequence to obtain low-frequency target motion component data; Step S2: Obtain operator IMU inertial data; Perform gait cycle recognition on the operator's IMU inertial data to obtain gait phase feature data; Perform variable gain image stabilization compensation on the original camera image sequence according to the gait phase feature data to obtain an image stabilization compensation image sequence; Step S3: acquiring multimodal operating environment data; estimating the operator's motion state based on the multimodal operating environment data to obtain operator motion state data; calculating the target object optical flow on the image stabilization compensation image sequence based on the operator motion state data to obtain a target motion vector field; Step S4: Evaluate the target motion vector field to obtain a target trajectory candidate point set; perform RANSAC outlier removal on the target trajectory candidate point set to obtain a valid target trajectory; perform physical constraint verification on the valid target trajectory to obtain the target's true motion trajectory; Step S5: extracting multiple features of the target object based on the target's real motion trajectory to obtain a target comprehensive feature vector; The target object is tracked by multi-target fusion according to the target comprehensive feature vector to obtain the target positioning and recognition result.

2. The intelligent camera positioning and identification method according to claim 1, characterized in that: Step S1 includes the following steps: Step S11: obtaining a camera original image sequence; performing pre-processing filtering on each image in the camera original image sequence to obtain a denoised camera image sequence; Step S12: performing continuous frame difference calculation on the denoised camera image sequence to obtain inter-frame motion difference data; Step S13: performing time domain sampling and reorganization on the denoised camera image sequence based on the inter-frame motion difference data to obtain a motion timing matrix of the image sequence; Step S14: performing fast Fourier transform on the image sequence motion time matrix to obtain image sequence spectrum data; Step S15: Calculating the spectrum energy density of the image sequence spectrum data to obtain the image spectrum energy distribution characteristics; Step S16: performing frequency band separation on the image sequence spectrum data according to the image spectrum energy distribution characteristics to obtain multi-band component data of the image sequence; Step S17: performing low-frequency target motion component identification on the multi-band component data of the image sequence to obtain low-frequency target motion component data.

3. The intelligent camera positioning and identification method according to claim 1, characterized in that: Step S2 includes the following steps: Step S21: The six-axis IMU sensor worn on the operator's head continuously collects three-axis acceleration and three-axis angular velocity data at a sampling frequency of 500 Hz, with the range set to ±16 g and ±2000° / s, to obtain the operator's IMU inertial data; Step S22: performing temperature drift compensation on the operator IMU inertial data to obtain corrected operator IMU inertial data; Step S23: performing complementary filtering fusion on the corrected operator IMU inertial data to obtain operator head posture angle data; Step S24: performing head motion spectrum transformation on the operator based on the operator's head posture angle data to obtain head motion spectrum feature data; Step S25: automatically identifying the operator's gait cycle based on the head motion spectrum feature data to obtain the operator's gait phase segmentation points; Step S26: extracting periodic features from the calibrated operator IMU inertial data based on the operator's gait phase segmentation points to obtain gait phase feature data; Step S27: obtaining the operator's plantar pressure signal; performing variable gain image stabilization compensation on the camera's original image sequence according to the gait phase feature data and the operator's plantar pressure signal to obtain a stabilized compensated image sequence.

4. The intelligent camera positioning and identification method according to claim 3, characterized in that: Step S27 includes the following steps: Step S271: obtaining the operator's plantar pressure signal; calculating the pressure center trajectory of the operator's plantar pressure signal to obtain plantar pressure gravity center data; Step S272: performing time series correlation modeling based on the gait phase feature data and the plantar pressure center of gravity data to obtain the operator's gait pressure correlation model; Step S273: Predicting the neck muscle tension based on the operator's gait pressure correlation model to obtain a prediction parameter for the operator's neck tremor; Step S274: constructing a variable gain control strategy based on the operator's neck tremor prediction parameters to obtain camera image stabilization compensation parameters; Step S275: performing mechanical image stabilization driving on the smart camera based on the camera image stabilization compensation parameter to obtain mechanical compensation displacement data; Step S276: performing electronic image stabilization fusion processing on the mechanical compensation displacement data to obtain camera fusion stabilization parameters; Step S277: performing real-time image stabilization processing on the camera original image sequence based on the camera fusion stabilization parameters to obtain a stabilized compensated image sequence.

5. The intelligent camera positioning and identification method according to claim 1, characterized in that: Step S3 includes the following steps: Step S31: By integrating an IMU gyroscope, a three-axis magnetometer, an ultrasonic ranging sensor, and a lidar sensor array, multi-dimensional perception data of the operator's surrounding environment is collected at a synchronous sampling frequency of 500 Hz to generate multi-modal operating environment data; Step S32: performing data alignment and calibration on the multimodal operating environment data to obtain synchronized multimodal operating environment data; Step S33: performing noise feature recognition on the synchronous multimodal operating environment data to obtain sensor noise feature parameters; Step S34: filtering the synchronized multimodal operating environment data based on the sensor noise characteristic parameters to obtain denoised multimodal operating environment data; Step S35: constructing a state vector for the denoised multimodal operating environment data to obtain an operator multimodal state vector; Step S36: performing Kalman filter parameter design based on the operator's multimodal state vector to obtain Kalman filter configuration parameters; performing Kalman filtering on the denoised multimodal operating environment data according to the Kalman filter configuration parameters to obtain operator motion state data; Step S37: Calculate the target object optical flow on the image stabilization compensation image sequence according to the operator motion state data to obtain a target motion vector field.

6. The intelligent camera positioning and identification method according to claim 5, characterized in that: Step S37 includes the following steps: Step S371: extracting motion pattern features from the operator motion state data to obtain an operator motion feature vector; performing motion pattern recognition modeling based on the operator motion feature vector to obtain an operator motion state classification model; Step S372: classifying the operator motion state data in real time according to the operator motion state classification model to obtain an operator motion state recognition result; Step S373: performing state transition statistics on the operator motion state recognition results to obtain an operator motion state transition probability table; Step S374: smoothing the operator motion state recognition result based on the operator motion state transition probability table to obtain the operator stable motion state data; Step S375: mapping visual algorithm parameters according to the operator's stable motion state data to obtain a visual algorithm parameter configuration mapping table; Step S376: adjusting the parameter configuration of the image stabilization compensation image sequence based on the visual algorithm parameter configuration mapping table to obtain the image sequence visual parameter configuration data; Step S377: performing pyramid optical flow calculation on the image stabilization compensation image sequence according to the image sequence visual parameter configuration data to obtain a target motion vector field.

7. The intelligent camera positioning and identification method according to claim 1, characterized in that: Step S4 includes the following steps: Step S41: performing vector field density calculation on the target motion vector field to obtain a motion vector density distribution map; Step S42: Preliminary screening of target candidate regions based on the motion vector density distribution map to obtain candidate target motion region data; Step S43: performing connectivity detection on the candidate target motion region data to obtain a connected target motion region label; Step S44: performing temporal association on the target motion vector field based on the connected target motion region labels to obtain an initial target trajectory candidate sequence; Step S45: performing stability screening on the initial target trajectory candidate sequence according to the low-frequency target motion component data to obtain a target trajectory candidate sequence; Step S46: performing trajectory smoothness evaluation on the target trajectory candidate sequence to obtain a trajectory smoothness evaluation index; performing quality screening on the target trajectory candidate sequence according to the trajectory smoothness evaluation index to obtain a target trajectory candidate point set; Step S47: Perform RANSAC outlier removal on the target trajectory candidate point set to obtain a valid target trajectory; perform physical constraint verification on the valid target trajectory to obtain the target's true motion trajectory.

8. The intelligent camera positioning and identification method according to claim 1, characterized in that: Step S5 includes the following steps: Step S51: positioning the target area bounding box based on the target's real motion trajectory to obtain target bounding box coordinate data; Step S52: cropping the target area of ​​the image stabilization compensation image sequence according to the target bounding box coordinate data to obtain a target image block sequence; Step S53: performing color histogram feature extraction on the target image block sequence to obtain a target color feature vector; Step S54: Calculate the HOG gradient directional feature based on the target image block sequence to obtain the target HOG feature vector; Step S55: performing LBP texture pattern recognition on the target image block sequence to obtain a target LBP feature vector; Step S56: performing feature vector concatenation based on the target color feature vector, the target HOG feature vector, and the target LBP feature vector to obtain a target comprehensive feature vector; Step S57: Perform multi-target fusion tracking on the target object according to the target comprehensive feature vector to obtain the target positioning and recognition result.

9. An intelligent camera positioning and recognition system, characterized in that: For executing the intelligent camera positioning and recognition method according to claim 1, the intelligent camera positioning and recognition system comprises: The image preprocessing module is used to obtain the original image sequence of the camera; perform multi-band decoupling on the original image sequence of the camera to obtain multi-band component data of the image sequence; perform low-frequency target motion component recognition on the multi-band component data of the image sequence to obtain low-frequency target motion component data; The image stabilization compensation module is used to obtain the operator's IMU inertial data; perform gait cycle recognition on the operator's IMU inertial data to obtain gait phase feature data; perform variable gain image stabilization compensation on the camera's original image sequence based on the gait phase feature data to obtain a stabilized compensated image sequence; A motion estimation module is used to obtain multimodal operating environment data; estimate the operator's motion state based on the multimodal operating environment data to obtain the operator's motion state data; and calculate the target object optical flow on the image stabilization compensation image sequence based on the operator's motion state data to obtain the target motion vector field; The trajectory evaluation and screening module is used to evaluate the target motion vector field and obtain the target trajectory candidate point set; perform RANSAC outlier removal on the target trajectory candidate point set to obtain the valid target trajectory; perform physical constraint verification on the valid target trajectory to obtain the target's true motion trajectory; The target recognition and tracking module is used to extract multiple features of the target object based on the target's real motion trajectory to obtain the target comprehensive feature vector; based on the target comprehensive feature vector, multi-target fusion tracking is performed on the target object to obtain the target positioning and recognition result.

10. A pair of smart glasses, characterized in that: The smart glasses include a processor, a memory, a camera, a display screen, a communication circuit, an audio component and a ranging component; the camera includes a visible light camera and an infrared camera; the memory stores a program that can be loaded by the processor and execute the smart camera positioning and identification method as described in any one of claims 1 to 8.

Images