Attitude data generation method, image acquisition equipment and storage medium

Abstract

The invention discloses an attitude data generation method, an image acquisition device and a storage medium, relates to the technical field of attitude detection, and discloses an attitude data generation method comprising the following steps: obtaining a dual-channel image, a dual-channel audio and dual-channel inertial sensing data collected by an earphone unit; extracting heterogeneous visual features based on the dual-channel image; extracting a spatial auditory attention mask based on the dual-channel audio; determining action characteristics based on the dual-channel inertial sensing data; fusing the heterogeneous visual features, the spatial auditory attention mask and the action features to obtain attitude description features; and according to the attitude description features, determining attitude evaluation indexes corresponding to the detection target. Through three-mode data fusion processing of visual, auditory and inertial sensing data, a semantic correspondence relationship is established among different modes, and a more stable and context-aware behavior recognition capability is provided, so that the accuracy of attitude anomaly detection is improved.

Description

Technical Field

[0001] The present application relates to the technical field of posture detection, and in particular to a method for generating posture data, an image acquisition device, and a storage medium. Background Art

[0002] In long-term office and study scenarios, health problems caused by improper posture, prolonged sitting and fatigue (such as cervical spondylosis, decreased vision, and lack of concentration) are becoming increasingly common.

[0003] Most existing fatigue monitoring and posture recognition solutions rely on sensors installed on detection devices, such as headphones, to detect and identify posture based on motion data. However, these detection methods are unable to accurately identify subtle changes in the user's posture, facial expressions, or static fatigue states, resulting in low posture detection accuracy.

[0004] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a method for generating posture data, an image acquisition device and a storage medium, aiming to solve the technical problem of low posture detection accuracy.

[0006] To achieve the above objectives, the present application proposes a method for generating posture data, which includes: Acquire dual-channel images, dual-channel audio, and dual-channel inertial sensor data collected by the headset unit; Extracting heterogeneous visual features based on the dual-channel image; extracting spatial auditory attention masks based on the dual-channel audio; and determining motion features based on the dual-channel inertial sensing data; fusing the heterogeneous visual features, the spatial auditory attention mask, and the action features to obtain a posture description feature; According to the posture description features, a posture evaluation index corresponding to the detection target is determined.

[0007] In one embodiment, before the step of fusing the heterogeneous visual features, the spatial auditory attention mask, and the action features to obtain a posture description feature, the method for generating posture data includes: Projecting the feature dimensions of the heterogeneous visual features, the feature dimensions of the spatial auditory attention mask, and the feature dimensions of the action features into a preset dimension; The step of fusing the heterogeneous visual features, the spatial auditory attention mask, and the action features to obtain a posture description feature includes: Determining a query condition, a key, and a value of a self-attention network, wherein when the visual feature is the query condition, the spatial auditory attention mask and the motion feature are both the key and the value; The query condition, the key, and the value are fused based on a preset attention algorithm to obtain the posture description feature.

[0008] In one embodiment, the step of extracting heterogeneous visual features based on the dual-channel image includes: Obtaining basic feature information corresponding to the two-channel images respectively, and determining the absolute value of the feature difference corresponding to the basic feature information; Calculate the separation convolution result of the absolute value of the feature difference, and compress the separation convolution result into the visual feature based on the average pooling layer.

[0009] In one embodiment, before the steps of extracting heterogeneous visual features based on the dual-channel image; extracting spatial auditory attention masks based on the dual-channel audio; and determining motion features based on the dual-channel inertial sensing data, the method for generating posture data further includes: The dual-channel image is corrected according to the dual-channel inertial sensing data and a preset Kalman filter algorithm.

[0010] In one embodiment, the step of extracting a spatial auditory attention mask based on the dual-channel audio includes: Determining a frequency domain transformation result of the dual-channel audio; Determining time delay information or sound intensity difference of the dual-channel audio after frequency domain transformation, and determining the direction of the sound source based on the time delay information or the sound intensity difference; According to the sound source direction and a preset mask algorithm, a spatial attention mask of the dual-channel audio after the frequency domain transformation is constructed.

[0011] In one embodiment, the step of determining motion characteristics based on the dual-channel inertial sensing data includes: Determining a sliding window length and a sliding step size of the dual-channel inertial sensing data; Extracting basic statistical features of the dual-channel inertial sensing data based on the sliding window length and the sliding step size; The basic statistical features are mapped to the action features.

[0012] In one embodiment, after the step of determining a posture evaluation index corresponding to the detection target based on the posture description feature, the method for generating posture data further includes: If the posture evaluation index is an abnormal index, generating a detection result indicating that the posture of the detection target is abnormal; A gesture prompt type is determined according to the detection result, and a gesture adjustment prompt is output based on the gesture prompt type.

[0013] In one embodiment, if the posture detection condition is met, the step of acquiring the dual-channel image, dual-channel audio, and dual-channel inertial sensing data collected by the earphone unit is performed, wherein the posture detection condition at least includes: The image acquisition device is in wearable state; Arrival of the preset detection period; The posture evaluation index is a normal index.

[0014] In addition, to achieve the above-mentioned purpose, the present application also proposes an image acquisition device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the method for generating posture data as described above.

[0015] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the method for generating posture data as described above are implemented.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: During posture data detection, the system first acquires dual-channel image, dual-channel audio, and dual-channel inertial sensor data from the earphone unit. It then extracts a spatial auditory attention mask from the dual-channel audio data, extracts heterogeneous visual features from the dual-channel image data, and extracts motion features from the inertial sensor data. The three-dimensional features are then fused to produce posture description features. Finally, the posture description features are used to determine the posture evaluation index corresponding to the detected target. Therefore, during posture detection, the system fuses the three modalities of visual, auditory, and inertial sensor data to establish semantic correspondences between the different modalities, providing more stable and context-aware behavior recognition capabilities, thereby improving the accuracy of posture anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 A flowchart of the first embodiment of the method for generating posture data of the present application is provided; Figure 2 A flowchart of the second embodiment of the method for generating posture data of the present application is provided; Figure 3 Flowchart of the third embodiment of the method for generating posture data of this application Figure 4 Schematic diagram of a brief flow of a method for generating posture data obtained by combining various embodiments of the present application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the method for generating posture data in the embodiment of the present application.

[0020] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0022] In long-term office and study scenarios, health problems caused by improper posture, prolonged sitting and fatigue (such as cervical spondylosis, decreased vision, and lack of concentration) are becoming increasingly common.

[0023] Most existing fatigue monitoring and posture recognition solutions rely on sensors installed on detection devices, such as headphones, to detect and identify posture based on motion data. However, these detection methods are unable to accurately identify subtle changes in the user's posture, facial expressions, or static fatigue states, resulting in low posture detection accuracy.

[0024] The main solution of the embodiment of the present application is to obtain dual-channel images, dual-channel audio and dual-channel inertial sensor data collected by the earphone unit; Extracting heterogeneous visual features based on the dual-channel image; extracting spatial auditory attention masks based on the dual-channel audio; and determining motion features based on the dual-channel inertial sensing data; fusing the heterogeneous visual features, the spatial auditory attention mask, and the action features to obtain a posture description feature; According to the posture description features, a posture evaluation index corresponding to the detection target is determined.

[0025] Specifically, during the posture detection process, through the fusion processing of three-modal data of vision, auditory and inertial sensing data, a semantic correspondence is established between different modalities, providing a more stable and context-aware behavior recognition capability, thereby improving the accuracy of posture anomaly detection.

[0026] It should be noted that the execution entity of this embodiment can be a computing service device with data processing, network communication, and program execution functions, or an image acquisition device capable of performing the aforementioned functions. Image acquisition devices include headsets equipped with cameras, such as TWS (True Wireless Stereo) headsets, and wearable devices equipped with video modules, such as smart home systems, online education systems, and security monitoring systems, that require image capture through video modules. This improves the user experience and market competitiveness of these devices in actual use, promoting the development of smart wearable technology. This embodiment and the following embodiments are described below using a headset equipped with cameras as an example. The headset has two independent cameras (left and right ear cameras) that can effectively capture high-precision depth (distance) information based on its inherent stereoscopic vision structure advantages, such as sufficient baseline length. Furthermore, the spatial positioning of feature points is more accurate and stable, significantly reducing the measurement error of monocular vision and improving the accuracy of pose angle estimation.

[0027] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0028] The present application embodiment provides a sitting posture detection method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the sitting posture detection method of the present application.

[0029] In this embodiment, the sitting posture detection method includes steps S10 to S40: Step S10: Acquire dual-channel images, dual-channel audio, and dual-channel inertial sensing data collected by the earphone unit.

[0030] In this embodiment, after the user wears the headphones, or for a period of time after wearing the headphones, the headphones perform real-time posture detection and synchronously collect three types of data through the cameras of the headphone units on both sides, the microphones on both sides, and the inertial measurement units on both sides, namely, synchronous collection of dual-channel images, dual-channel audio, and dual-channel inertial sensing data. Among them, these data are usually collected synchronously to ensure the uniformity of the data in the time dimension and avoid errors in the detection data. In addition to the camera, the headphones are also equipped with an inertial sensor (IMU, Inertial Measurement Unit), where the inertial sensor usually includes a gyroscope, an accelerometer, and a magnetometer, which can collect three-axis angular velocity, linear acceleration data, and geomagnetic data.

[0031] It's important to note that in addition to extracting data from both sides of the earphones, posture detection can also be performed based on one earphone unit, extracting image data, audio data, and inertial sensor data from a single side. Extracting dual-channel image, audio, and inertial sensor data and performing correction analysis based on the data from both sides improves the accuracy of the generated posture data compared to extracting data from only one side.

[0032] Step S20: extracting heterogeneous visual features based on the dual-channel image; extracting spatial auditory attention masks based on the dual-channel audio; and determining action features based on the dual-channel inertial sensing data.

[0033] In this embodiment, heterogeneous visual features are extracted from images, including the user's head posture at the current moment and facial fatigue motion features, such as blinking and yawning. Spatial auditory attention masks play a key role in identifying fatigue sound features such as yawning and sighing in audio data. Motion features are inertial feature vectors that represent the user's motion state.

[0034] In this embodiment, after the headphones collect multi-dimensional data, the feature extraction module extracts the spatial auditory attention mask from the dual-channel audio, extracts the heterogeneous visual features of the dual-channel image, and extracts the motion features of the inertial sensor data. The feature extraction modules corresponding to different features can be the same or different. When extracting features from the above-mentioned different dimensional data, the extraction modules can perform feature extraction simultaneously or sequentially, which is not limited in this application.

[0035] By extracting spatial auditory attention masks, visual features, and motion features, we can perform fusion matching between features to extract related features, such as the image and sound when the user yawns, and the inertial data when the head posture changes when yawning. We then fuse the multi-dimensional data to improve the accuracy of posture detection.

[0036] Optionally, the above features can also be extracted through a pre-trained neural network model.

[0037] Step S30 , fusing the heterogeneous visual features, the spatial auditory attention mask, and the action features to obtain a posture description feature.

[0038] In this embodiment, cross-modal fusion can be performed through a cross-modal fusion module such as a Transformer network, including setting one of the features as the Query of the Transformer network, that is, the main reference direction, and setting the other two features as the Key and Value of the Transformer network, so as to perform feature attention fusion, thereby achieving more comprehensive and accurate posture recognition and fatigue state detection. Among them, the posture description feature is the feature after the fusion of feature data of three different dimensions, and the Transformer network can output the posture evaluation index of the detection target based on the feature. Among them, before the feature fusion is performed, the feature can be uniformly projected in dimension.

[0039] Optionally, the visual features and spatial auditory attention masks extracted by the two feature extraction modules can be input into the cross-modal fusion module, and the features can be fused through the cross-modal feature fusion module. Finally, the motion features from the inertial sensor are fused to obtain the posture evaluation index.

[0040] Step S40: determining a posture evaluation index corresponding to the detection target according to the posture description feature.

[0041] In this embodiment, the posture evaluation index is the output of the cross-modal fusion module and is used to evaluate the current posture information. The posture evaluation index includes at least posture categories for human posture recognition, such as sitting, standing, moving, lowering the head, raising the head, tilting the head, etc., a fatigue score that quantifies the degree of physical and mental fatigue in the user's current posture, and a sedentary probability that reflects the detection target's long-term low-energy consumption behavior. In addition to the parameters listed above, other indicators for posture detection may also be included, and this application does not make too many restrictions here.

[0042] For example, in the visual features, the user yawns, and the spatial auditory attention mask is used to identify the presence of the yawning sound. At the same time, the inertial sensor data contains the motion features caused by yawning. At this time, after the posture description features are obtained through feature fusion processing, based on the posture description features such as the probability of each feature, the weight ratio, etc., it is generated that the user is in a state of lowering his head, the fatigue score is 0.8 points (0-1, the fatigue level and the score are positively correlated), and the probability of sitting for a long time is 80%.

[0043] This embodiment fuses spatial auditory attention masks, visual features, and motion features to obtain a posture evaluation index representing the user's actual posture, thereby achieving more comprehensive and accurate posture recognition and fatigue state detection.

[0044] Furthermore, in this embodiment, when posture detection conditions are met, the processing action of step S10 is executed. The posture detection conditions include the image acquisition device, i.e., the headphones, being in a wearing state. For example, when the user first puts on the headphones, the headphones trigger the detection process and collect data from three dimensions in real time. Optionally, detection begins after a preset detection period, such as 30 minutes after wearing the headphones. Optionally, if the posture evaluation index meets the posture normality index, indicating that the user's posture is normal during the current detection cycle, detection proceeds to the next detection cycle.

[0045] This embodiment provides a method for generating posture data. After the user wears headphones or the headphones enter the next detection cycle, three-dimensional data on both sides of the headphones are collected in real time and synchronously, and vector features of these data are obtained. Subsequently, vector features of different dimensions are fused to obtain a posture evaluation index for evaluating the user's current posture. In this way, the accuracy of posture anomaly detection is improved by constructing a correlation between different modalities.

[0046] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above introduction and will not be described in detail later. Figure 2 After step S40, steps S50 to S60 are also included: Step S50: If the posture evaluation index is an abnormal index, a detection result indicating that the posture of the detection target is abnormal is generated.

[0047] In this embodiment, the headset uses a threshold judgment to analyze whether the posture evaluation index is an abnormal indicator. Among them, the state evaluation index includes at least posture category, fatigue score and sedentary probability; the abnormal indicator refers to the fatigue score and / or sedentary probability parameters corresponding to the posture category, which meet the indicator parameters of abnormal posture. In the judgment process, for example, if the score value of the fatigue score in the posture evaluation index is lower than the preset score, it means that the posture evaluation index meets the abnormal indicator, and a posture abnormal state detection result can be generated at this time. Similarly, if the sedentary probability is greater than the preset probability, a posture abnormal state detection result can also be generated. Among them, if the above two types of posture parameters simultaneously meet the parameters of the posture abnormality evaluation index, the posture abnormality detection result of the detection target can be directly generated.

[0048] Optionally, if the posture evaluation index does not meet the abnormal index, that is, the index is a normal index, it means that the user's current posture and state are normal. At this time, data collection can be continued, that is, entering the data collection and analysis processing of the next detection cycle.

[0049] Step S60: determining a posture prompt type according to the detection result, and outputting a posture adjustment prompt based on the posture prompt type.

[0050] In this embodiment, in addition to setting a corresponding threshold judgment mechanism, the headphones also trigger a health feedback module when the current state is determined to be abnormal, for example, by playing spatial audio and reminding the user in the form of voice prompts. Therefore, the posture prompt type can be determined based on the detection results, and then the posture adjustment prompt can be output based on the posture prompt type. For example, if the abnormal detection result is that the user is in a state of fatigue or has been sitting for a long time, the prompts that can be generated at this time may include "You seem a little tired, please take a break" or "You have been sitting for 45 minutes, please move around appropriately." These prompts are played in the form of headphone spatial audio to enhance directional perception and immersion. At the same time, the headphones will automatically record the behavioral status, fatigue level and user feedback at the current time point to form a health log for subsequent analysis and health management. In addition, posture adjustment methods can also be output in other ways, which are not limited in this application.

[0051] This embodiment provides a method for generating posture data. After fusing vector features of different dimensions to obtain a posture evaluation index for evaluating the user's current posture, the posture evaluation index is analyzed. When the posture evaluation index is an abnormal index, a detection result of the user's posture abnormality is generated. At the same time, a corresponding prompt is output based on the detection result. In this way, abnormal posture detection is achieved by fusing data of multiple dimensions, thereby improving the accuracy of posture abnormality detection.

[0052] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above introduction and will not be repeated hereafter. On this basis, features of different dimensions are fused through the neural network of the Transformer architecture. Before fusion, the feature dimensions of the heterogeneous visual features, the feature dimensions of the spatial auditory attention mask, and the feature dimensions of the action features need to be projected into preset dimensions so that these features are aligned in the same feature space to facilitate subsequent fusion processing. Therefore, please refer to Figure 3 , step S30 includes steps S31~S32: Step S31, determine the query conditions, keys and values of the self-attention network.

[0053] In this embodiment, in the Transformer network, Query, Key, and Value are the core components of the attention mechanism, and the three work together to achieve dynamic information focusing and integration of the input sequence. Therefore, the query conditions, keys, and values of the self-attention network can be determined. In the feature fusion process, when one of the features is a Query, the other two features are fused and spliced as Key and Value. Therefore, when the visual feature is the query condition, the spatial auditory attention mask and the action feature are both keys and values. For example, the visual feature is the query condition Query. At this time, the Key and Value are formed by splicing the spatial auditory attention mask and the action feature to form a 256-dimensional fusion vector, which is then converted to 128 dimensions through linear mapping to ensure consistency with the dimension of the Query vector in the Transformer attention mechanism, which is convenient for trimodal joint modeling. It can be understood that when one of the features is a query condition, the other two features are the key and value spliced together.

[0054] Step S32: fusing the query condition, the key, and the value based on a preset attention algorithm to obtain the posture description feature.

[0055] In this embodiment, the preset attention algorithm is as follows: , Among them, Attention (Q, K, V) is the preset attention algorithm, K T is the transposed matrix of key K, and 128 is the feature dimension. The output of this algorithm is a posture description feature. Based on this state description feature, the Transformer-based neural network can further generate outputs including posture classification, fatigue score (range 0-1), and sedentary state probability (range 0-1). Therefore, posture evaluation indicators include at least one of fatigue score, posture classification, and sedentary probability. This allows posture analysis to be performed using multi-dimensional indicator data, improving posture detection effectiveness.

[0056] Exemplarily, the module for fusing multiple features is a VACT (Visual - Audio - IMUCross-modal Transformer) trimodal fusion module, wherein the dimension of the visual feature is 128, the dimension of the spatial auditory attention mask after flattening is 196 dimensions (14×14), and the dimension of the motion feature is 128. Based on this, the trimodal fusion module is used to fuse the features and output the posture evaluation result corresponding to the posture description feature. The processing steps include: Step 1: Unify the feature dimension projection and project the spatial auditory attention mask from 196 dimensions to 128 dimensions in the fully connected layer so that the three modal inputs are all 128.

[0057] In step 2, cross-modal Transformer fusion is performed. The Transformer parameter configuration includes the number of stacking layers: 2 layers, the number of multi-head attention: 4, and the dimension of attention features in each layer: 128 dimensions. In the query-key-value setting, the query is the visual feature, and the key and value are the spatial auditory attention mask + motion feature (the concatenation of the two). The fused features are then calculated based on the Transformer attention mechanism formula.

[0058] Step 3: Real-time posture and fatigue state prediction output. In the fully connected output structure, the input dimension is 128-dimensional posture description features, and the output indicators include at least posture classification (head down, head up, head sideways, etc.), fatigue level score (range 0-1), and sedentary state probability (range 0-1). The fatigue score and sedentary probability can be calculated using activation functions such as the Sigmoid activation function. In the final output stage, the user's real-time posture classification result, fatigue score (continuous value), and sedentary state probability (probability value) are output.

[0059] This embodiment provides a method for generating posture data, which performs feature fusion processing through a vision-auditory-IMU three-modal deep fusion architecture, thereby significantly improving the accuracy and reliability of behavior recognition and fatigue monitoring when performing posture detection and fatigue detection based on headphones, and solving the problem of low posture detection accuracy.

[0060] Based on the first embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the first embodiment above can be referred to the above introduction and will not be repeated hereafter. On this basis, after obtaining the dual-channel image, the basic features corresponding to the dual-channel images can be extracted through the feature extraction network, and then the visual difference between the two, that is, the absolute difference of the feature difference between the basic features, is calculated, and then the convolution separation result of the absolute value of the feature difference is calculated, and finally the separation convolution result is compressed into a visual feature vector of a preset dimension through the average pooling layer.

[0061] Specifically, the SHVE (Stereo-Heterogeneous Visual Encoder) heterogeneous visual feature extraction module can be used to process heterogeneous images on both sides. After inputting the left and right ear images L and R, the two images are first passed through a convolutional neural network to extract basic features. Then, the visual difference map (the absolute value of the visual difference) is calculated using the formula: ΔV = |f(L) - f(R)|. is the absolute value of visual difference, is the basic feature of the left image, The 128-dimensional feature vector is the basis for the right image. This captures the difference between the left and right images, providing a basis for subsequent analysis of posture changes and facial micro-movements. Next, depthwise convolution and global average pooling (GAP) are performed to output a 128-dimensional difference feature vector. This feature vector is highly sensitive to head posture changes, such as lowering and raising the head, and facial micro-movements (such as blinking and yawning), effectively capturing subtle changes in fatigue and attention.

[0062] For example, the module input includes: left and right real-time RGB image frames, with a uniform image size of 224×224×3. The module structure and detailed algorithm steps include: Step 1: Image preprocessing: unify the image size to 224×224 (Bilinear interpolation), and then perform pixel normalization, dividing each channel by 255, and the range is [0,1].

[0063] Step 2: Perform feature extraction using a basic CNN (Convolutional Neural Network). The CNN architecture is MobileNetV3-Small, and the output feature dimension is 14×14×128.

[0064] Step 3: Calculate heterogeneous difference convolution features, and the CNN feature map is recorded as: , Then calculate the heterogeneous visual difference map, that is, the absolute value of the feature difference between the two: , The output dimension is 14×14×128.

[0065] Step 4: Spatial feature encoding: Separate convolution and feature pooling are performed based on a network with a depthwise separable convolution type, a convolution kernel size of 3×3, a stride of 1, padding of the same, an activation function of ReLU6, and a feature pooling method of Global Average Pooling (GAP) to obtain a visual feature vector of the preset dimension.

[0066] It should be noted that the above parameters are only used for explanation and are not intended to limit the present application.

[0067] Optionally, after capturing the original image, since the user's head will inevitably shake when wearing headphones, in order to prevent these slight movements from interfering with recognition, it is necessary to first perform motion compensation processing on the dual-channel image. Specifically, the headset processing system can use the Kalman filter algorithm to stabilize and correct the image based on the angular velocity changes in the inertial sensor data. Therefore, before step S20, it is necessary to correct the dual-channel image based on the inertial sensor data and the preset Kalman filter algorithm to ensure that the image still has high recognition accuracy under micro-motion conditions.

[0068] This embodiment provides a method for generating posture data by extracting visual features of heterogeneous images and performing feature fusion processing based on the heterogeneous images on both sides, thereby capturing the difference information between the left and right images, thereby providing a basis for subsequent posture changes and facial micro-movement analysis, thereby improving the accuracy of posture detection.

[0069] Based on the first embodiment of the present application, in the fifth embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to above and will not be described in detail. On this basis, in addition to processing the heterogeneous images of the earphone units on both sides, it is also necessary to process the audio collected by the earphones on both sides so as to extract features based on the difference in the audio between the two sides.

[0070] Based on this, as an optional implementation method for obtaining a spatial auditory attention mask vector, the frequency domain transformation result of the dual-channel audio in the speech data can be determined, and then the time delay information or sound intensity difference of the dual-channel audio after the frequency domain transformation can be determined. Based on the time delay information or sound intensity difference, the direction of the sound source can be determined, and finally, the spatial attention mask of the dual-channel audio after the frequency domain transformation can be constructed according to the sound source direction.

[0071] Specifically, the SAAM (Spatial Auditory Attention Module) processes speech data. Taking two-channel audio as input, the module performs a short-time Fourier transform (SFT) on the audio, converting the sound signal from the time domain to the frequency domain. Parameters are set to a 25 millisecond frame length, a 10 millisecond frame shift, and 512 FFT points. Next, the GCC-PHAT algorithm performs generalized cross-correlation on the signals between the two channels to calculate the time delay or intensity difference of the sound, thereby inferring the spatial direction of the current sound. Based on this directional information, a spatial mask is constructed using a Gaussian function with a sigmoid function of σ² = 0.25. The resulting 14×14 attention map is output. This attention map highlights audio information related to fatigue (such as yawning, long sighs, and respiratory rate changes), providing an important SAT mask for subsequent multimodal fusion processing.

[0072] For example, the module input includes a real-time captured two-channel audio stream with a sampling rate of 16kHz. The module structure and detailed algorithm steps are as follows: Step 1: Perform a short-time Fourier transform (STFT) on the two-channel audio data based on a frame length of 25 ms (400 samples), a frame shift of 10 ms (160 samples), and an FFT count of 512 points.

[0073] Step 2: Calculate spatial sound difference characteristics using the TDOA algorithm: GCC-PHAT - Maximum time difference: ±0.6ms (corresponding to an approximately 20cm time interval) - Sound direction accuracy: ±1 degree (θ from -90° to +90°) to calculate time delay information or sound intensity difference.

[0074] Step 3: Calculate the spatial auditory attention mask. The mask calculation formula is as follows: , in, is the mask, i, j are index variables, The incident angle of the (i, j)th array element, is the angular difference between the current position (i, j) and the target sound source direction, corresponding to the sound source direction, IID is the binaural sound intensity difference, and 0.25 is the Sigmoid function σ².

[0075] The final output mask size is 14×14 (matching the visual features).

[0076] It should be noted that the above parameters are only used for explanation and are not intended to limit the present application.

[0077] This embodiment provides a method for generating posture data by extracting auditory attention mask information from the voice data collected on both sides of the earphone. This mask plays a key role in identifying fatigue sound features such as yawning and sighing, thereby improving the accuracy of posture detection after subsequent feature fusion.

[0078] Based on the first embodiment of the present application, in the sixth embodiment of the present application, the same or similar contents as those in the first embodiment above can be referred to the above introduction and will not be repeated hereafter. On this basis, the motion features corresponding to the inertial sensor data can be extracted through the IMU motion feature extraction module. In this process, the sliding window length and sliding step length of the dual-channel inertial sensor data can be determined first. For example, the sliding window is 2 seconds and the sliding step length is 1 second, that is, 2 seconds of data (approximately 200 points) are taken each time and slide once every 1 second. Then, based on the sliding window length and sliding compensation, the basic statistical features of the dual-channel inertial sensor data are extracted. These basic statistical features include at least one of the mean (Mean), standard deviation (Std), maximum value (Max), zero crossing rate (ZCR) and variation rate (Variation Rate). Finally, the basic statistical features are mapped to motion features.

[0079] Specifically, the input parameters of the IMU motion feature extraction module are the three-axis acceleration and angular velocity signals in the inertial sensor data, with a sampling frequency of 100 Hz. The algorithm steps of this module include: Step 1: Window division, set a sliding window length of 2 seconds (200 data points) and a sliding step size of 1 second.

[0080] Step 2: Feature extraction. In the action feature extraction module, the following statistical features are extracted for each channel: The mean, standard deviation, maximum value, zero-crossing rate, and rate of change are calculated, with a total dimension of 6 channels × 5 features = 30 dimensions. This is then projected to a 128-dimensional representation using a fully connected layer.

[0081] Step 3: Feature normalization and buffering. Smooth the current output using the historical sliding window results, and finally output the IMU motion feature vector (128 dimensions).

[0082] It should be noted that the above parameters are only used for explanation and are not intended to limit the present application.

[0083] For example, in order to help understand the implementation process of the method for generating posture data obtained by combining the present embodiment with the above embodiments, please refer to Figure 4 , Figure 4 A brief flowchart of a method for generating posture data is provided, specifically: After the user puts on the TWS earphones, the earphones start the detection system, and then the earphones synchronously collect trimodal data, including left and right images, binaural audio, and three-axis IMU data. The image jitter information is then corrected through the IMU motion compensation module and Kalman filtering. Then the three feature extraction modules perform feature extraction at the same time. Among them, the IMU motion feature extraction module maps the inertial sensor data FC to a 128-dimensional vector based on a 2-second sliding window, 6 channels, and 5 features to obtain motion features; the SAAM spatial auditory attention module uses the STFT+GCC-PHAT method to generate a mask in the spatial direction based on the sound source direction (TDOA) or sound intensity difference IID, and obtains a 14×14 mask map, which is the spatial auditory attention mask; at the same time, the SHVE heterogeneous visual feature extraction module uses the convolutional neural network MobileNetV3 to extract the basic features fL and fR of the left and right images, and calculates the absolute value of the feature difference of the basic feature information. , then convolution and pooling are performed through Depthwise+GAP to finally obtain 128-dimensional visual features.

[0084] The VACT cross-modal fusion module then fuses the three-dimensional module data. The input data consists of 128-dimensional visual features, auditory mask features, and IMU motion features. The feature dimensions are first unified, and then fused through a two-layer Transformer × 4-head attention process. The resulting output is a posture assessment value, including fatigue score, posture classification (head down / head up / sedentary), and silent time statistics. Finally, the posture assessment value is checked to see if it exceeds a threshold. If not, the next frame or time period is tested. If so, the health feedback module issues a prompt, such as triggering a TWS spatial audio and voice prompt, outputting the message "You seem tired. Please take a break." Finally, a reminder log and user feedback are recorded. Based on this, the binaural ectopic vision-auditory-IMU three-modal deep fusion architecture is used to achieve the fusion of multi-dimensional data, which improves the accuracy and reliability of behavior recognition and fatigue monitoring. Combined with the characteristics of TWS headphones being light, concealed and sustainable to wear, it can be quickly integrated into existing smart headphone hardware without the user having to change their usage habits, thereby improving the accuracy of posture detection.

[0085] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the method for generating posture data of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0086] The present application provides an image acquisition device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for generating posture data in the above-mentioned first embodiment.

[0087] Reference below Figure 5 , which shows a structural schematic diagram of an image acquisition device suitable for implementing the embodiments of the present application. Figure 5 The image acquisition device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0088] like Figure 5 As shown, the image acquisition device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the image acquisition device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007, such as a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008, such as a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003, such as a magnetic tape or hard disk; and communication device 1009. The communication device 1009 can allow the image acquisition device to communicate with other devices wirelessly or wired to exchange data. Although the figure shows an image acquisition device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems can be implemented or provided instead.

[0089] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are performed.

[0090] The image acquisition device provided in this application, employing the method for generating posture data in the above-described embodiment, can resolve the technical problem of low posture detection accuracy. Compared to the prior art, the image acquisition device provided in this application has the same beneficial effects as the method for generating posture data in the above-described embodiment. Other technical features of this image acquisition device are the same as those disclosed in the above-described embodiment and are not further described here.

[0091] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0092] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0093] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, and the computer-readable program instructions are used to execute the method for generating posture data in the above-mentioned embodiment.

[0094] The computer-readable storage medium provided herein may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including, but not limited to, wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0095] The computer-readable storage medium may be included in the image acquisition device, or may exist independently without being assembled into the image acquisition device.

[0096] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the image acquisition device, the image acquisition device is caused to: acquire dual-channel images, dual-channel audio, and dual-channel inertial sensor data acquired by the earphone unit; Extracting heterogeneous visual features based on the dual-channel image; extracting spatial auditory attention masks based on the dual-channel audio; and determining motion features based on the dual-channel inertial sensing data; fusing the heterogeneous visual features, the spatial auditory attention mask, and the action features to obtain a posture description feature; According to the posture description features, a posture evaluation index corresponding to the detection target is determined.

[0097] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0098] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the module, program segment or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0099] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0100] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned method for generating posture data. This computer-readable storage medium can address the technical issue of low posture detection accuracy. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are similar to those of the method for generating posture data provided in the aforementioned embodiments, and are not further elaborated here.

[0101] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for generating posture data, characterized in that: The method for generating the posture data comprises: Acquire dual-channel images, dual-channel audio, and dual-channel inertial sensor data collected by the headset unit; Extracting heterogeneous visual features based on the dual-channel image; extracting spatial auditory attention masks based on the dual-channel audio; and determining motion features based on the dual-channel inertial sensing data; fusing the heterogeneous visual features, the spatial auditory attention mask, and the action features to obtain a posture description feature; According to the posture description features, a posture evaluation index corresponding to the detection target is determined.

2. The method for generating posture data according to claim 1, wherein: Before the step of fusing the heterogeneous visual features, the spatial auditory attention mask, and the action features to obtain a posture description feature, the method for generating posture data further includes: Projecting the feature dimensions of the heterogeneous visual features, the feature dimensions of the spatial auditory attention mask, and the feature dimensions of the action features into a preset dimension; The step of fusing the heterogeneous visual features, the spatial auditory attention mask, and the action features to obtain a posture description feature includes: Determining a query condition, a key, and a value of a self-attention network, wherein when the visual feature is the query condition, the spatial auditory attention mask and the motion feature are both the key and the value; The query condition, the key, and the value are fused based on a preset attention algorithm to obtain the posture description feature.

3. The method for generating posture data according to claim 1, wherein: The step of extracting heterogeneous visual features based on the dual-channel image comprises: Obtaining basic feature information corresponding to the two-channel images respectively, and determining the absolute value of the feature difference corresponding to the basic feature information; Calculate the separation convolution result of the absolute value of the feature difference, and compress the separation convolution result into the visual feature based on the average pooling layer.

4. The method for generating posture data according to claim 3, wherein: extracting heterogeneous visual features based on the dual-channel image; extracting a spatial auditory attention mask based on the dual-channel audio; Before the step of determining the motion characteristics based on the dual-channel inertial sensing data, the method for generating posture data further includes: The dual-channel image is corrected according to the dual-channel inertial sensing data and a preset Kalman filter algorithm.

5. The method for generating posture data according to claim 1, wherein: The step of extracting a spatial auditory attention mask based on the dual-channel audio comprises: Determining a frequency domain transformation result of the dual-channel audio; Determining time delay information or sound intensity difference of the dual-channel audio after frequency domain transformation, and determining the direction of the sound source based on the time delay information or the sound intensity difference; According to the sound source direction and a preset mask algorithm, a spatial attention mask of the dual-channel audio after the frequency domain transformation is constructed.

6. The method for generating posture data according to claim 1, wherein: The step of determining motion characteristics based on the dual-channel inertial sensing data includes: Determining a sliding window length and a sliding step size of the dual-channel inertial sensing data; Extracting basic statistical features of the dual-channel inertial sensing data based on the sliding window length and the sliding step size; The basic statistical features are mapped to the action features.

7. The method for generating posture data according to claim 1, wherein: After the step of determining the posture evaluation index corresponding to the detection target according to the posture description feature, the method for generating posture data further includes: If the posture evaluation index is an abnormal index, generating a detection result indicating that the posture of the detection target is abnormal; A gesture prompt type is determined according to the detection result, and a gesture adjustment prompt is output based on the gesture prompt type.

8. The method for generating posture data according to any one of claims 1 to 7, characterized in that: If the posture detection condition is met, executing the acquisition of the dual-channel image, dual-channel audio, and dual-channel inertial sensor data collected by the earphone unit; The step of: wherein the posture detection condition at least includes: The image acquisition device is in wearable state; Arrival of the preset detection period; The posture evaluation index is a normal index.

9. An image acquisition device, characterized in that: The image acquisition device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the method for generating posture data according to any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the method for generating posture data according to any one of claims 1 to 8 are implemented.

Images