Pose data generation method, image acquisition device and storage medium

Abstract

The application discloses a posture data generation method, an image acquisition device and a storage medium, relates to the technical field of posture detection, and discloses a posture data generation method, which comprises the following steps: acquiring double-channel images, double-channel audio and double-channel inertial sensing data collected by a headset unit; extracting heterogeneous visual features based on the double-channel images; extracting a spatial auditory attention mask based on the double-channel audio; determining motion features based on the double-channel inertial sensing data; fusing the heterogeneous visual features, the spatial auditory attention mask and the motion features to obtain posture description features; and determining posture evaluation indexes corresponding to a detection target according to the posture description features. Through three-mode data fusion processing of visual, auditory and inertial sensing data, semantic corresponding relationships are established between different modes, more stable and context-aware behavior recognition capabilities are provided, and the accuracy of posture anomaly detection is improved.

Description

Technical Field

[0001] This application relates to the field of attitude detection technology, and in particular to methods for generating attitude data, image acquisition equipment, and storage media. Background Technology

[0002] In long-term office and study settings, health problems caused by improper posture, prolonged sitting, and fatigue (such as cervical spondylosis, decreased vision, and difficulty concentrating) are becoming increasingly common.

[0003] Most solutions for fatigue monitoring and posture recognition involve placing sensors on detection devices such as headphones, and then performing posture detection and recognition based on the motion data from these sensors. However, this type of detection method cannot accurately identify subtle changes in user posture, facial expressions, and static fatigue states, resulting in low posture detection accuracy.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a method for generating posture data, an image acquisition device, and a storage medium, in order to solve the technical problem of low posture detection accuracy.

[0006] To achieve the above objectives, this application proposes a method for generating attitude data, the method comprising: Acquire dual-channel images, dual-channel audio, and dual-channel inertial sensing data from the headphone unit; Heterogeneous visual features are extracted based on the dual-channel images; spatial auditory attention masks are extracted based on the dual-channel audio; and motion features are determined based on the dual-channel inertial sensing data. By fusing the heterogeneous visual features, the spatial auditory attention mask, and the motion features, a posture description feature is obtained. Based on the posture description features, determine the posture evaluation index corresponding to the detection target.

[0007] In one embodiment, prior to the step of fusing the heterogeneous visual features, the spatial auditory attention mask, and the action features to obtain the pose description features, the method for generating the pose data includes: The feature dimensions of the heterogeneous visual features, the feature dimensions of the spatial auditory attention mask, and the feature dimensions of the action features are projected as preset dimensions. The step of fusing the heterogeneous visual features, the spatial auditory attention mask, and the motion features to obtain the posture description features includes: Determine the query conditions, keys, and values ​​of the self-attention network, wherein when the visual feature is the query condition, the spatial auditory attention mask and the action feature are both the keys and the values; The pose description features are obtained by fusing the query conditions, the key, and the value based on a preset attention algorithm.

[0008] In one embodiment, the step of extracting heterogeneous visual features based on the dual-channel image includes: Obtain the basic feature information corresponding to the dual-channel images respectively, and determine the absolute value of the feature difference corresponding to the basic feature information; The absolute value of the feature difference is calculated for the separable convolution result, and the separable convolution result is compressed into the visual feature based on the average pooling layer.

[0009] In one embodiment, prior to the steps of extracting heterogeneous visual features based on the dual-channel image; extracting a spatial auditory attention mask based on the dual-channel audio; and determining motion features based on the dual-channel inertial sensing data, the method for generating the pose data further includes: The dual-channel image is corrected based on the dual-channel inertial sensing data and a preset Kalman filter algorithm.

[0010] In one embodiment, the step of extracting spatial auditory attention mask based on the dual-channel audio includes: Determine the frequency domain transformation result of the dual-channel audio; Determine the time delay information or sound intensity difference of the dual-channel audio after frequency domain transformation, and determine the direction of the sound source based on the time delay information or the sound intensity difference; Based on the sound source direction and a preset masking algorithm, a spatial attention mask for the dual-channel audio after frequency domain transformation is constructed.

[0011] In one embodiment, the step of determining motion characteristics based on the dual-channel inertial sensing data includes: Determine the sliding window length and sliding step size of the dual-channel inertial sensing data; Based on the sliding window length and the sliding step size, the basic statistical features of the dual-channel inertial sensing data are extracted. The basic statistical features are mapped to the action features.

[0012] In one embodiment, after the step of determining the pose evaluation index corresponding to the detection target based on the pose description features, the method for generating pose data further includes: If the posture evaluation index is an abnormal index, generate a detection result of the posture abnormality of the target. The attitude prompt type is determined based on the detection results, and an attitude adjustment prompt is output based on the attitude prompt type.

[0013] In one embodiment, if the attitude detection conditions are met, the step of acquiring dual-channel images, dual-channel audio, and dual-channel inertial sensing data collected by the headphone unit is performed; wherein the attitude detection conditions include at least: The image acquisition device is in the wearing state; The preset detection period has arrived; The attitude evaluation index is a normal index.

[0014] In addition, to achieve the above objectives, this application also proposes an image acquisition device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the attitude data generation method described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the attitude data generation method described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: In the posture data detection process, dual-channel images, dual-channel audio, and dual-channel inertial sensing data are first acquired from the headphone unit. Then, a spatial auditory attention mask is extracted from the dual-channel audio data, heterogeneous visual features are extracted from the dual-channel images, and motion features are extracted from the inertial sensing data. These three dimensions of features are then fused to obtain posture description features. Finally, the posture evaluation index corresponding to the detected target is determined using these posture description features. Therefore, in the posture detection process, the fusion of visual, auditory, and inertial sensing data across three modalities establishes semantic correspondences between different modalities, providing more stable and context-aware behavior recognition capabilities, thereby improving the accuracy of posture anomaly detection. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating the first embodiment of the method for generating attitude data in this application; Figure 2 A flowchart illustrating the second embodiment of the method for generating attitude data in this application; Figure 3 A flowchart illustrating the third embodiment of the attitude data generation method of this application. Figure 4 This is a simplified flowchart illustrating the method for generating attitude data obtained by combining various embodiments of this application. Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the attitude data generation method in the embodiments of this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] In long-term office and study settings, health problems caused by improper posture, prolonged sitting, and fatigue (such as cervical spondylosis, decreased vision, and difficulty concentrating) are becoming increasingly common.

[0023] Most solutions for fatigue monitoring and posture recognition involve placing sensors on detection devices such as headphones, and then performing posture detection and recognition based on the motion data from these sensors. However, this type of detection method cannot accurately identify subtle changes in user posture, facial expressions, and static fatigue states, resulting in low posture detection accuracy.

[0024] The main solution of this application embodiment is: to acquire dual-channel images, dual-channel audio, and dual-channel inertial sensing data collected by the headphone unit; Heterogeneous visual features are extracted based on the dual-channel images; spatial auditory attention masks are extracted based on the dual-channel audio; and motion features are determined based on the dual-channel inertial sensing data. By fusing the heterogeneous visual features, the spatial auditory attention mask, and the motion features, a posture description feature is obtained. Based on the posture description features, determine the posture evaluation index corresponding to the detection target.

[0025] Specifically, during the posture detection process, the three-modal data fusion processing of visual, auditory and inertial sensing data is used to establish semantic correspondence between different modalities, providing a more stable and context-aware behavior recognition capability, thereby improving the accuracy of posture anomaly detection.

[0026] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, or an image acquisition device capable of performing the above functions. The image acquisition device includes earphones with cameras, such as TWS (True Wireless Stereo) earphones, and wearable devices such as smart home systems, online education systems, and security monitoring systems that require image capture via video modules. This improves the user experience and market competitiveness of these devices in practical use, and promotes the development of smart wearable technology. The following description uses an earphone with a camera as an example to illustrate this embodiment and the following embodiments. This earphone has two independent cameras (left and right ear cameras), which can effectively capture high-precision depth (distance) information based on its inherent stereo vision structure advantages, such as sufficient baseline length. Simultaneously, the spatial positioning of feature points is more accurate and stable, significantly reducing the measurement error of monocular vision and improving the accuracy of attitude angle estimation.

[0027] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0028] This application provides a sitting posture detection method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the sitting posture detection method of this application.

[0029] In this embodiment, the sitting posture detection method includes steps S10 to S40: Step S10: Acquire dual-channel images, dual-channel audio, and dual-channel inertial sensing data collected by the headphone unit.

[0030] In this embodiment, after the user puts on the headphones, or for a period of time after putting on the headphones, the headphones perform real-time attitude detection and simultaneously collect three types of data through the cameras on both sides of the headphone unit, the microphones on both sides, and the inertial measurement units on both sides: simultaneously collecting dual-channel images, dual-channel audio, and dual-channel inertial sensor data. These data are typically collected simultaneously to ensure data consistency over time and avoid detection errors. In addition to the cameras, the headphones are equipped with inertial sensors (IMUs), which typically include gyroscopes, accelerometers, and magnetometers, capable of collecting three-axis angular velocity, linear acceleration data, and geomagnetic data.

[0031] It should be noted that, in addition to extracting data from both sides of the earphone, attitude detection can also be performed based on one earphone unit, that is, extracting image data, audio data, and inertial sensing data from one side. Specifically, by extracting dual-channel image, audio, and inertial sensing data, and performing correction analysis based on data from both sides, the accuracy of the generated attitude data is improved compared to extracting data from only one side.

[0032] Step S20: Extract heterogeneous visual features based on the dual-channel image; extract a spatial auditory attention mask based on the dual-channel audio; and determine motion features based on the dual-channel inertial sensing data.

[0033] In this embodiment, heterogeneous visual features are user head posture features extracted from images, including the user's current head posture and facial fatigue movement features, such as blinking and yawning. Spatial auditory attention masking plays a crucial role in recognizing fatigue sound features such as yawning and sighing in sound data. Motion features are inertial feature vectors representing the user's motion state.

[0034] In this embodiment, after the headphones collect multi-dimensional data, the spatial auditory attention mask in the dual-channel audio is extracted by the feature extraction module, the heterogeneous visual features of the dual-channel images are extracted, and the motion features of the inertial sensing data are extracted. The feature extraction modules corresponding to different features can be the same or different. When extracting features from the above-mentioned data of different dimensions, the extraction module can perform feature extraction simultaneously or sequentially; this application does not impose any limitations on this.

[0035] By extracting spatial auditory attention masks, visual features, and motion features, we can perform feature fusion and matching to extract related features, such as images and sounds of a user yawning and inertial data when the head posture changes during yawning. We can then fuse multi-dimensional data to improve the accuracy of posture detection.

[0036] Alternatively, the above features can also be extracted using a pre-trained neural network model.

[0037] Step S30: Fuse the heterogeneous visual features, the spatial auditory attention mask, and the action features to obtain posture description features.

[0038] In this embodiment, cross-modal fusion can be performed using a cross-modal fusion module such as a Transformer network. This involves setting one feature as the Query (primary reference direction) of the Transformer network, and setting the other two features as the Key and Value of the Transformer network, thereby performing attention fusion of features to achieve more comprehensive and accurate pose recognition and fatigue state detection. The pose description feature is a fusion of feature data from three different dimensions, and the Transformer network can output a pose evaluation index of the detected target based on this feature. Before feature fusion, the features can undergo dimensional uniform projection processing.

[0039] Optionally, the visual features extracted by the two feature extraction modules and the spatial auditory attention mask can be input into the cross-modal fusion module, and the features can be fused through the cross-modal feature fusion module. Finally, the motion features from the inertial sensor are fused to obtain the attitude evaluation index.

[0040] Step S40: Determine the posture evaluation index corresponding to the detection target based on the posture description features.

[0041] In this embodiment, the posture evaluation index is the output of the cross-modal fusion module, used to evaluate the current posture information. The posture evaluation index includes at least the posture categories for identifying human postures, such as sitting, standing, moving, looking down, looking up, looking up, and tilting the head, a fatigue score that quantifies the user's physical and mental fatigue level in the current posture, and the probability of prolonged sitting that reflects the target's ability to maintain low-energy consumption behavior for an extended period. In addition to the parameters listed above, other posture detection indicators may also be included, which are not limited in this application.

[0042] For example, in visual features, if a user yawns, the spatial auditory attention mask can be used to identify the sound of the yawn. At the same time, the inertial sensing data contains the motion features generated by the yawn. After obtaining the posture description features through feature fusion processing, based on the posture description features such as the probability and weight of each feature, it can be generated that the user is looking down, has a fatigue score of 0.8 (0-1, the degree of fatigue is positively correlated with the score), and has a probability of sitting for a long time of 80%.

[0043] This embodiment integrates spatial auditory attention masking, visual features, and motion features to obtain a posture evaluation index that represents the user's actual posture, thereby achieving more comprehensive and accurate posture recognition and fatigue detection.

[0044] Further, in this embodiment, when the posture detection conditions are met, the processing action of step S10 is executed. The posture detection conditions include the image acquisition device, i.e., the headphones, being in a wearing state. For example, when a user first puts on the headphones, the headphones trigger the detection process and collect data in three dimensions in real time. Optionally, detection begins after a preset detection period, such as 30 minutes after wearing the headphones. Optionally, if the posture evaluation index meets the normal posture index, it indicates that the user's posture is normal within the current detection cycle, and the next detection cycle is then performed.

[0045] This embodiment provides a method for generating posture data. After the user wears headphones or the headphones enter the next detection cycle, data from three dimensions on both sides of the headphones are collected in real time and synchronously, and the vector features of these data are obtained. Then, the vector features of different dimensions are fused to obtain a posture evaluation index for evaluating the user's current posture. In this way, by constructing the correlation between different modalities, the accuracy of posture anomaly detection is improved.

[0046] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 After step S40, steps S50 to S60 are also included: Step S50: If the attitude evaluation index is an abnormal index, generate the detection result of the abnormal attitude of the detection target.

[0047] In this embodiment, the headphones use threshold judgment to analyze whether the posture evaluation indicators are abnormal. These indicators include at least posture category, fatigue score, and probability of prolonged sitting. Abnormal indicators refer to the parameters of the fatigue score and / or probability of prolonged sitting corresponding to a posture category, which satisfy the parameters of an abnormal posture. During the judgment process, for example, if the fatigue score in the posture evaluation indicators is lower than a preset score, it indicates that the posture evaluation indicators meet the abnormal indicators, and a state detection result for an abnormal posture can be generated. Similarly, if the probability of prolonged sitting is greater than a preset probability, a state detection result for an abnormal posture can also be generated. If both types of posture parameters simultaneously satisfy the parameters of the posture abnormality evaluation indicators, a posture abnormality detection result for the detected target can be directly generated.

[0048] Optionally, if the posture evaluation index does not meet the abnormal index, that is, the index is a normal index, it means that the user's current posture and state are normal. At this time, data collection can continue, that is, the data collection and analysis processing of the next detection cycle can begin.

[0049] Step S60: Determine the attitude prompt type based on the detection result, and output an attitude adjustment prompt based on the attitude prompt type.

[0050] In this embodiment, in addition to setting a corresponding threshold judgment mechanism, the headphones also trigger a health feedback module when an abnormal state is detected. For example, it plays spatial audio and reminds the user in the form of voice prompts. Therefore, the posture prompt type can be determined based on the detection result, and then a posture adjustment prompt can be output according to the posture prompt type. For example, if the abnormal detection result is that the user is in a fatigued state or has been sitting for a long time, the prompts that can be generated include "You seem a little tired, please take a rest" or "You have been sitting for 45 minutes, please move around a bit." These prompts are played in the form of spatial audio from the headphones, enhancing orientation awareness and immersion. At the same time, the headphones will also automatically record the behavioral state, fatigue level, and user response at the current time point to form a health log for subsequent analysis and health management. In addition, posture adjustment methods can also be output in other ways, which are not limited here.

[0051] This embodiment provides a method for generating posture data. After fusing vector features of different dimensions to obtain a posture evaluation index for assessing the user's current posture, the posture evaluation index is analyzed. When the posture evaluation index is an abnormal index, a detection result of the user's posture abnormality is generated. At the same time, a corresponding prompt is output based on the detection result. In this way, abnormal posture detection is achieved by fusing data from multiple dimensions, thereby improving the accuracy of posture abnormality detection.

[0052] Based on the first embodiment of this application, in the third embodiment of this application, the same or similar content as the first embodiment can be referred to the above description, and will not be repeated hereafter. On this basis, a neural network with a Transformer architecture is used to fuse features from different dimensions. Before fusion, the feature dimensions of heterogeneous visual features, the feature dimensions of the spatial auditory attention mask, and the feature dimensions of action features need to be projected to a preset dimension so that these features are aligned in the same feature space, facilitating subsequent fusion processing. Therefore, please refer to... Figure 3 Step S30 includes steps S31 to S32: Step S31: Determine the query conditions, keys, and values ​​for the self-attention network.

[0053] In this embodiment, in the Transformer network, Query, Key, and Value are the core components of the attention mechanism, working together to achieve dynamic information focusing and integration of the input sequence. Therefore, the query condition, key, and value of the self-attention network can be determined. During feature fusion, when one feature is a Query, the other two features are fused and concatenated to form the Key and Value. Therefore, when a visual feature is a query condition, both the spatial auditory attention mask and the action feature serve as the key and value. For example, if the visual feature is the query condition, the Key and Value are formed by concatenating the spatial auditory attention mask and the action feature to form a 256-dimensional fusion vector, which is then linearly mapped to 128 dimensions to ensure consistency with the Query vector in the Transformer attention mechanism, facilitating trimodal joint modeling. It can be understood that when one feature is a query condition, the other two features are the key and value formed by concatenating features.

[0054] Step S32: Based on a preset attention algorithm, the query conditions, the key, and the value are fused to obtain the pose description features.

[0055] In this embodiment, the preset attention algorithm is as follows: , Where Attention(Q,K,V) is the preset attention algorithm, and K T Let K be the transpose of the key K, and 128 be the feature dimension. The algorithm outputs a posture description feature. Based on this state description feature, the Transformer architecture neural network can further generate outputs including posture classification, fatigue score (range 0~1), and sedentary probability (range 0~1). Therefore, posture evaluation metrics should include at least one of fatigue score, posture classification, and sedentary probability to improve posture detection performance through multi-dimensional metric data.

[0056] For example, the module for fusing multiple features is the VACT (Visual-Audio-IMUCross-modal Transformer) trimodal fusion module, where the visual features have a dimension of 128, the spatial auditory attention mask has a dimension of 196 (14×14) after flattening, and the action features have a dimension of 128. Based on this, the processing steps for fusing features through the trimodal fusion module and outputting the pose evaluation result corresponding to the pose description features include: Step 1: Unify feature dimension projection. In the fully connected layer, project the spatial auditory attention mask from 196 dimensions to 128 dimensions so that all three modal inputs are 128.

[0057] Step 2: Perform cross-modal Transformer fusion. The Transformer parameter configuration includes the number of stacked layers: 2, the number of multi-head attention: 4, and the dimension of attention features per layer: 128. In the Query-Key-Value setting, Query is the visual feature, and Key and Value are the spatial auditory attention mask + action features (features after the two are concatenated). Then, the fused features are calculated based on the Transformer attention mechanism formula.

[0058] Step 3: Real-time posture and fatigue state prediction output. In the fully connected output structure, the input dimension is 128-dimensional posture description features, and the output indicators include at least posture classification (head down, head up, head sideways, etc.), fatigue level score (range 0 ~ 1), and sedentary state probability (range 0 ~ 1). The fatigue score and sedentary probability can be calculated using activation functions such as the Sigmoid activation function. In the final output stage, the user's real-time posture classification result, fatigue score (continuous value), and sedentary state probability (probability value) are output.

[0059] This embodiment provides a method for generating posture data. By using a visual-auditory-IMU trimodal deep fusion architecture to fuse features, the accuracy and reliability of behavior recognition and fatigue monitoring are significantly improved when using headphones for posture detection and recognition and fatigue detection, thus solving the problem of low posture detection accuracy.

[0060] Based on the first embodiment of this application, in the fourth embodiment of this application, the same or similar content as the first embodiment can be referred to the above description, and will not be repeated hereafter. On this basis, after obtaining the dual-channel image, the basic features corresponding to the dual-channel image can be extracted by a feature extraction network, and then the visual difference between the two, i.e., the absolute difference of the feature difference between the basic features, is calculated. Then, the convolution separation result of the absolute value of the feature difference is calculated, and finally, the separation convolution result is compressed into a visual feature vector of a preset dimension through an average pooling layer.

[0061] Specifically, the heterogeneous visual feature extraction module SHVE (Stereo - Heterogeneous Visual Encoder) can be used to process the bilateral heterogeneous images. After inputting the left and right ear images L and R, the bilateral images are first processed by a convolutional neural network to extract basic features. Then, the visual difference map, i.e., the absolute value of visual difference, is calculated using the formula: ΔV = |f(L) - f(R)|. The absolute value of visual difference. The basic features of the left image, This forms the foundational feature vector for the right-hand image. This method captures the differences between the left and right images, providing a basis for subsequent posture changes and facial micro-movement analysis. Next, through depthwise convolution and global average pooling (GAP) operations, a 128-dimensional difference feature vector is finally output. This feature vector is highly sensitive to head posture changes, head tilting, head raising, and facial micro-movements (such as blinking and yawning), effectively capturing subtle changes in fatigue and attention.

[0062] For example, the module input includes: left and right real-time RGB image frames, with a uniform image size of 224×224×3. The module structure and detailed algorithm steps include: Step 1: Image preprocessing. The image size is unified to 224×224 (Bilinear interpolation). Then, pixel normalization is performed, with each channel divided by 255, ranging from [0,1].

[0063] Step 2: Feature extraction is performed using a basic CNN (Convolutional Neural Network). The CNN network structure is MobileNetV3-Small, and the output feature dimension is 14×14×128.

[0064] Step 3: Calculate heterogeneous differential convolutional features. The CNN feature map is denoted as: , Then, the heterogeneous visual difference map is calculated, which is the absolute value of the feature difference between the two: , The output dimension is 14×14×128.

[0065] Step 4: Spatial feature encoding. Based on a network with Depthwise Separable Convolution, kernel size 3×3, stride=1, padding=same, activation function ReLU6, and feature pooling method GlobalAverage Pooling (GAP), perform separate convolution and feature pooling to obtain a visual feature vector of the preset dimension.

[0066] It should be noted that the above parameters are for illustrative purposes only and are not intended to limit this application.

[0067] Optionally, after acquiring the original image, since users inevitably experience head movements when wearing headphones, motion compensation processing is required for the dual-channel images to avoid interference from these slight movements in recognition. Specifically, the headphone processing system can use the Kalman filter algorithm to stabilize and correct the image based on the angular velocity changes in the inertial sensing data. Therefore, before step S20, the dual-channel images need to be corrected based on the inertial sensing data and the preset Kalman filter algorithm to ensure that the images still have high recognition accuracy under slight movement conditions.

[0068] This embodiment provides a method for generating posture data. By extracting visual features from heterogeneous images and performing feature fusion processing on both sides of the heterogeneous images, the difference information between the left and right images is captured, thereby providing a basis for subsequent posture change and facial micro-movement analysis, thus improving the accuracy of posture detection.

[0069] Based on the first embodiment of this application, in the fifth embodiment of this application, the same or similar content as the first embodiment can be referred to the above description, and will not be repeated hereafter. In addition to processing the heterogeneous images of the two earphone units, it is also necessary to process the audio collected by the two earphones in order to extract features based on the differences in the audio from both sides.

[0070] Based on this, as an optional implementation of obtaining spatial auditory attention mask vectors, the frequency domain transformation result of dual-channel audio in speech data can be determined, then the time delay information or sound intensity difference of the frequency domain transformed dual-channel audio can be determined, and the sound source direction can be determined based on the time delay information or sound intensity difference. Finally, a spatial attention mask of the frequency domain transformed dual-channel audio can be constructed based on the sound source direction.

[0071] Specifically, the speech data is processed using the Spatial Auditory Attention Module (SAAM). This module takes dual-channel audio as input and performs a short-time Fourier transform (SFT) to convert the sound signal from the time domain to the frequency domain. The parameters are set to a frame length of 25 milliseconds, a frame shift of 10 milliseconds, and 512 FFT points. Next, the GCC-PHAT algorithm is used to perform generalized cross-correlation processing on the signals between the two channels to calculate the time delay or intensity difference of the sound, thereby inferring the spatial direction of the current sound. Based on this directional information, a spatial mask is constructed using a Gaussian function, where the sigmoid function σ² = 0.25. The final output is a 14×14 attention map, which highlights fatigue-related audio information (such as yawning, sighing, and changes in breathing frequency), providing an important spatial auditory attention mask for subsequent multimodal fusion processing.

[0072] For example, the module input includes a real-time acquired dual-channel audio stream with a sampling rate of 16kHz. The module structure and detailed algorithm steps are as follows: Step 1: Perform a Short Time Fourier Transform (STFT) on the dual-channel audio data based on frame length: 25ms (400 sampling points) - frame shift: 10ms (160 sampling points) - FFT points: 512 points.

[0073] Step 2: Calculate spatial sound difference characteristics using the TDOA algorithm: GCC-PHAT - maximum time difference: ±0.6ms (corresponding to an approximate 20cm time interval) - sound direction accuracy: ±1 degree (θ from -90° to +90°), etc., to calculate time delay information or sound intensity difference.

[0074] Step 3: Perform spatial auditory attention mask calculation. The formula for this mask calculation is shown below: , in, Let i and j be the mask, and j be the index variables. The incident angle of the (i, j)th array element. IID represents the angular difference between the current position (i, j) and the direction of the target sound source, corresponding to the direction of the sound source. IID represents the binaural sound intensity difference, and 0.25 represents the Sigmoid function σ².

[0075] The final output mask size is 14×14 (matching visual features).

[0076] It should be noted that the above parameters are for illustrative purposes only and are not intended to limit this application.

[0077] This embodiment provides a method for generating posture data. By extracting auditory attention mask information from the voice data collected from both sides of the headphones, the mask plays a key role in recognizing fatigue sound features such as yawning and sighing, thereby improving the accuracy of posture detection after subsequent feature fusion.

[0078] Based on the first embodiment of this application, in the sixth embodiment of this application, the same or similar content as the first embodiment can be referred to the above description, and will not be repeated hereafter. On this basis, the motion features corresponding to the inertial sensing data can be extracted using the IMU motion feature extraction module. In this process, the sliding window length and sliding step size of the dual-channel inertial sensing data can be determined first. For example, the sliding window is 2 seconds, and the sliding step size is 1 second, meaning that 2 seconds of data (approximately 200 points) are taken each time, and the sliding occurs once every 1 second. Subsequently, based on the sliding window length and sliding compensation, the basic statistical features of the dual-channel inertial sensing data are extracted. These basic statistical features include at least one of the following: mean, standard deviation, maximum value, zero crossover rate (ZCR), and variation rate. Finally, the basic statistical features are mapped to motion features.

[0079] Specifically, the input parameters of the IMU motion feature extraction module are the triaxial acceleration and angular velocity signals from the inertial sensing data, with a sampling frequency of 100Hz. The algorithm steps of this module include: Step 1: Window division, set a sliding window with a length of 2 seconds (200 data points), and a sliding step of 1 second.

[0080] Step 2: Feature extraction. In the action feature extraction module, the following statistical features are extracted for each channel: The mean, standard deviation, maximum value, zero crossover rate, and rate of change have a total dimension of 30 (6 channels × 5 features). This is then projected using a fully connected layer to a 128-dimensional representation.

[0081] Step 3: Feature standardization and buffering. The current output is smoothed by using historical sliding window results. Finally, the IMU action feature vector (128 dimensions) is output.

[0082] It should be noted that the above parameters are for illustrative purposes only and are not intended to limit this application.

[0083] For example, to help understand the implementation flow of the attitude data generation method obtained by combining the above embodiments, please refer to... Figure 4 , Figure 4 A simplified flowchart illustrating a method for generating attitude data is provided, specifically: After a user puts on the TWS earbuds, the earbuds activate the detection system and then simultaneously collect trimodal data, including left and right side images, binaural audio, and triaxial IMU data. The IMU motion compensation module and Kalman filter then correct image jitter. Next, three feature extraction modules simultaneously extract features. The IMU motion feature extraction module maps the inertial sensing data FC to a 128-dimensional vector using a 2-second sliding window, 6 channels, and 5 features to obtain motion features. The SAAM spatial auditory attention module generates a spatial mask based on the sound source direction (TDOA) or sound intensity difference IID using the STFT+GCC-PHAT method, resulting in a 14×14 mask image, which serves as the spatial auditory attention mask. Simultaneously, the SHVE heterogeneous visual feature extraction module extracts the basic features fL and fR from the left and right side images using the MobileNetV3 convolutional neural network and calculates the absolute value of the feature difference between the basic feature information. Then, convolution and pooling are performed using Depthwise+GAP to obtain 128-dimensional visual features.

[0084] Next, the VACT cross-modal fusion module fuses the data from three dimensions. The input data consists of 128-dimensional visual features, auditory mask features, and IMU motion features. First, the feature dimensions are unified. Then, after feature fusion processing using a 2-layer Transformer × 4-head attention architecture, the outputs fatigue score, posture classification, and posture evaluation values ​​for the probability of prolonged sitting. Following this, the posture evaluation values ​​are output for state prediction, including determining the fatigue score, posture classification (head down / head up / prolonged sitting, etc.), and silent time statistics. Finally, it checks if the posture evaluation values ​​exceed a threshold. If not, it continues to detect data in the next frame or the next time period. If they exceed the threshold, a prompt is given based on the health feedback module, such as triggering TWS spatial audio + voice prompts, outputting a message like "You seem a little tired, please take a rest." Finally, the reminder log and user feedback information are recorded. Based on this, a deep fusion architecture of binaural visual-auditory-IMU trimodal is used to achieve the fusion of multi-dimensional data, which improves the accuracy and reliability of behavior recognition and fatigue monitoring. Combined with the lightweight, discreet and continuous wear characteristics of TWS earphones, it can be quickly integrated into existing smart earphone hardware without requiring users to change their usage habits, thus improving the accuracy of posture detection.

[0085] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the method of generating attitude data in this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0086] This application provides an image acquisition device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the attitude data generation method in the first embodiment described above.

[0087] The following is for reference. Figure 5 It shows a schematic diagram of the structure of an image acquisition device suitable for implementing the embodiments of this application. Figure 5 The image acquisition device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0088] like Figure 5 As shown, the image acquisition device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the image acquisition device. The processing unit 1001, the ROM 1002, and the RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the image acquisition device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows image acquisition devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0089] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0090] The image acquisition device provided in this application, employing the posture data generation method in the above embodiments, can solve the technical problem of low posture detection accuracy. Compared with the prior art, the beneficial effects of the image acquisition device provided in this application are the same as those of the posture data generation method provided in the above embodiments, and other technical features in this image acquisition device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0091] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0092] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0093] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the attitude data generation method in the above embodiments.

[0094] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM, or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0095] The aforementioned computer-readable storage medium may be included in the image acquisition device; or it may exist independently and not assembled into the image acquisition device.

[0096] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the image acquisition device, cause the image acquisition device to: acquire dual-channel images, dual-channel audio, and dual-channel inertial sensing data acquired by the headphone unit; Heterogeneous visual features are extracted based on the dual-channel images; spatial auditory attention masks are extracted based on the dual-channel audio; and motion features are determined based on the dual-channel inertial sensing data. By fusing the heterogeneous visual features, the spatial auditory attention mask, and the motion features, a posture description feature is obtained. Based on the posture description features, determine the posture evaluation index corresponding to the detection target.

[0097] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0098] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0099] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0100] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described attitude data generation method, thereby solving the technical problem of low attitude detection accuracy. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the attitude data generation method provided in the above embodiments, and will not be repeated here.

[0101] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for generating attitude data, characterized in that, The method for generating the attitude data includes: Acquire dual-channel images, dual-channel audio, and dual-channel inertial sensing data from the headphone unit; Extracting heterogeneous visual features based on the dual-channel images includes obtaining basic feature information corresponding to the dual-channel images respectively, and determining the absolute value of the feature difference corresponding to the basic feature information; calculating the decoupling convolution result of the absolute value of the feature difference, and compressing the decoupling convolution result into the heterogeneous visual features based on an average pooling layer; Extracting a spatial auditory attention mask based on the dual-channel audio includes determining the frequency domain transformation result of the dual-channel audio; determining the time delay information or sound intensity difference of the frequency-domain transformed dual-channel audio, and determining the sound source direction based on the time delay information or the sound intensity difference; and constructing a spatial attention mask of the frequency-domain transformed dual-channel audio based on the sound source direction and a preset masking algorithm. Determining motion features based on the dual-channel inertial sensing data includes determining the sliding window length and sliding step size of the dual-channel inertial sensing data; extracting basic statistical features of the dual-channel inertial sensing data based on the sliding window length and the sliding step size; and mapping the basic statistical features to the motion features. The feature dimensions of the heterogeneous visual features, the feature dimensions of the spatial auditory attention mask, and the feature dimensions of the action features are projected as preset dimensions. By fusing the heterogeneous visual features, the spatial auditory attention mask, and the action features, a posture description feature is obtained, including determining the query conditions, keys, and values ​​of the self-attention network, wherein when the visual feature is the query condition, the spatial auditory attention mask and the action feature are both the key and the value; the posture description feature is obtained by fusing the query conditions, the key, and the value based on a preset attention algorithm. Based on the posture description features, determine the posture evaluation index corresponding to the detection target.

2. The method for generating attitude data as described in claim 1, characterized in that, The heterogeneous visual features are extracted based on the dual-channel image; Based on the aforementioned dual-channel audio extraction spatial auditory attention mask; Before the step of determining motion features based on the dual-channel inertial sensing data, the method for generating the attitude data further includes: The dual-channel image is corrected based on the dual-channel inertial sensing data and a preset Kalman filter algorithm.

3. The method for generating attitude data as described in claim 1, characterized in that, After the step of determining the posture evaluation index corresponding to the detection target based on the posture description features, the method for generating posture data further includes: If the posture evaluation index is an abnormal index, generate a detection result of the posture abnormality of the target. The attitude prompt type is determined based on the detection results, and an attitude adjustment prompt is output based on the attitude prompt type.

4. The method for generating attitude data as described in any one of claims 1 to 3, characterized in that, If the attitude detection conditions are met, the process of acquiring the dual-channel image, dual-channel audio, and dual-channel inertial sensing data collected by the headphone unit is executed. The steps, wherein the attitude detection conditions include at least: The image acquisition device is in the wearing state; The preset detection period has arrived; The attitude evaluation index is a normal index.

5. An image acquisition device, characterized in that, The image acquisition device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the method for generating attitude data as described in any one of claims 1 to 4.

6. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the attitude data generation method as described in any one of claims 1 to 4.

Images