Implementation method and system of intelligent earphone interrupt function based on positioning technology

Abstract

The application discloses an implementation method and system of an intelligent earphone interrupt function based on a positioning technology. The method comprises the following steps: establishing a connection between an intelligent earphone and a mobile host terminal and synchronizing a clock, and determining a relative position of multiple parties in space; calculating a distance difference by receiving sound data, and solving three-dimensional coordinates and distance information of a sound source based on a hyperbolic positioning principle; determining an interrupt priority value in combination with a semantic analysis result of the sound data, and sending an intelligent reminder to a user according to the interrupt priority value. The application realizes accurate spatial perception and intelligent importance judgment of surrounding sound events, enables the intelligent earphone to actively provide an interrupt type reminder at a critical moment, and improves the intelligence and practicability of human-computer interaction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent earphone interaction, and more particularly, to an implementation method and system of an intelligent earphone interrupt function based on positioning technology. BACKGROUND

[0002] As the core category of wearable devices, the development trend of intelligent earphones is evolving from passive response to active environmental perception and intelligent interaction. Among them, the "intelligent interrupt" function aims to filter out information that is really important to the user from the complex surrounding environment and remind the user, which is one of the key technologies to improve user experience. The implementation of this function highly depends on the accurate perception and understanding of the sound source event, and the core challenge lies in how to accurately answer the two questions of "what event has occurred" and "where the event has occurred".

[0003] To achieve the above goal, there are mainly two technical paths in the prior art:

[0004] First, a sound source positioning scheme based on a dual-ear microphone array. This scheme uses the microphones built-in the left and right earphones to form a microphone array with a very short physical baseline, and estimates the horizontal azimuth angle of the sound source by calculating the time difference (TDOA) and intensity difference (IID) of the sound signal arriving at the two ears. However, this scheme has inherent defects: 1. It severely lacks the positioning ability in the vertical dimension. Since the dual-ear connection is usually parallel to the ground, this array cannot effectively perceive the change in the elevation angle of the sound source in the vertical direction, resulting in that its positioning information is incomplete two-dimensional information, which cannot achieve accurate positioning in three-dimensional space. For example, the system cannot distinguish sound sources from the same direction but different floors. 2. It is limited by the extremely short distance between the earphones (baseline), and its distance measurement accuracy will sharply decrease at several meters away, with very low reliability.

[0005] Second, a cooperative positioning scheme based on multiple devices (left earphone, right earphone, mobile host). In order to make up for the short baseline of the dual-ear, some existing technologies try to introduce the user's mobile host (such as a smart phone) as a third microphone node, and use the larger geometry formed by the three to perform three-dimensional sound source positioning, in order to restore the vertical dimension information. However, this scheme faces a key problem that has been long ignored: the relative spatial positions between the mobile host and the left and right earphones are random and dynamically changing (such as the phone may be held in the hand, placed in the pocket or on the table). The existing scheme usually assumes that the three devices are in a known, fixed geometry, which is seriously inconsistent with the actual use scenario. Since the exact spatial relationship between the three devices cannot be accurately obtained in real time, even if a three-node positioning model is used, the input basic parameters (node coordinates) themselves are inaccurate, resulting in that the final sound source positioning solution, including the distance and three-dimensional coordinates, has error layer by layer, and in some cases, the detection result is even more error-prone than the dual-ear scheme.

[0006] At the semantic analysis level, the above-mentioned sound source positioning accuracy is insufficient, resulting in the subsequent intelligent decision-making losing reliable basis. The system can understand the semantics of the sound (such as greeting or car horn) through the audio recognition algorithm, but because it cannot judge whether the sound source is from one meter away or ten meters away, the decision to interrupt it often does not match the urgency of the real scene, causing false positives, false negatives, and the opposite effect.

[0007] Therefore, there is an urgent need in the art for an intelligent earphone system and method that can accurately perceive and deeply integrate the perceived information with semantic information to fundamentally solve the problem of intelligent interruption inaccuracy caused by the lack or error of spatial context in the prior art.

[0008] The information disclosed in the Background section of the present invention is only intended to deepen the understanding of the general background of the present invention, and should not be regarded as acknowledging or implying in any form that the information constitutes prior art known to those skilled in the art. SUMMARY

[0009] The present invention proposes an implementation method and system of intelligent earphone interruption function based on positioning technology, which can realize accurate positioning of surrounding sound sources and intelligent semantic analysis, thereby providing intelligent interruption reminders to users at critical moments.

[0010] In a first aspect, the present invention provides an implementation method of intelligent earphone interruption function based on positioning technology, which comprises the following steps:

[0011] S100, establishing the connection between the left intelligent earphone, the right intelligent earphone and the mobile host terminal through the connection protocol, adjusting the clock offset of the left earphone and the right earphone to align with the master clock of the mobile host terminal, and determining the spatial relative positions of the left earphone, the right earphone and the mobile host terminal with the midpoint of the left earphone and the right earphone as the origin.

[0012] S110, the left earphone, the right earphone and the mobile host terminal respectively receive sound data, calculate the time difference between each two of the three devices, convert it into distance difference Δd based on the current sound speed rl , Δd lp , Δd rp , and substitute it into the hyperbolic equation set to solve the three-dimensional (X, Y, Z) coordinates of the sound source by a nonlinear optimization algorithm, and determine the distance information between the sound source and the mobile host terminal based on the coordinate values;

[0013] S120, performing semantic analysis of the current scene based on the sound data and the distance information, determining an interruption priority value based on the semantic analysis result, and the interruption priority value representing the importance of the corresponding sound data;

[0014] S130, issuing a reminder to the user based on the sound data with the highest interrupt priority value and greater than a preset threshold.

[0015] Preferably, a Cartesian coordinate system is established with the midpoint of the left earphone and the right earphone as the origin, the earphone baseline direction as the x-axis, the horizontal and vertical direction perpendicular to the x-axis as the y-axis, and the vertical direction perpendicular to the x-axis and the y-axis as the z-axis; the initial coordinates of the left earphone are (-0.09, 0, 0), and the initial coordinates of the right earphone are (0.09, 0, 0); the relative coordinates (Xp, Yp, Zp) of the mobile host end are determined by the sensor.

[0016] Preferably, the sensor is a Bluetooth sensor or a UWB sensor, and the method comprises the following steps: (1) when it is detected that the spatial position of the mobile host end is relatively stable, a probe data packet is sent to the left earphone and the right earphone respectively; (2) the left earphone and the right earphone return a response data packet immediately after receiving the probe data packet, and the mobile host end records the round-trip time tl, tr and the data processing time tc of the earphone obtained from the response data packet after receiving the response data packet; (3) the distance d lp =(tl-tc)×c / 2, where c is the speed of light; the distance d rp =(tr-tc)×c / 2, where c is the speed of light; the initial coordinates of the left earphone, d lp the initial coordinates of the right earphone, and d rp the initial coordinates of the right earphone are established to obtain the circular relative position range of the mobile host end; and (4) the habit position of the user's mobile host end is preset, and the final spatial position relative coordinates (Xp, Yp, Zp) of the mobile host end are determined in combination with the circular relative position range.

[0017] Preferably, the sensor is an IMU sensor, and the method comprises the following steps: (1) assuming that the initial position of the mobile host end is (0, -1, 0), a constraint condition is set; (2) the spatial positions of the left earphone, the right earphone and the mobile host end are updated according to the IMU measurement values of the three; the spatial positions of the left earphone, the right earphone and the mobile host end are optimized according to the constraint condition; (3) it is judged whether the spatial positions of the three converge in a specific spatial range, and if so, the spatial positions of the three are output, and if not, step (2) is executed cyclically.

[0018] Preferably, first, the spectral features of different sounds are determined from the sound spectrum graph, and a sound data classification and recognition model of different sound data is trained from a sound spectrum feature set; second, the sound type is determined according to the current sound data and the trained sound data classification and recognition model, and if the sound type corresponds to a specific meaning, the interrupt priority value is calculated in combination with the distance information; if the sound type is a voice event, the voice event is converted into text for semantic recognition, and the interrupt priority value is calculated according to the semantic recognition result and the distance information.

[0019] In a second aspect, the embodiments of the present disclosure provide an implementation system of an intelligent earphone interrupt function based on a positioning technology, comprising:

[0020] A device connection and positioning module is configured to establish a connection between a left intelligent earphone, a right intelligent earphone and a mobile host terminal through a connection protocol, adjust a clock offset of the left earphone and the right earphone to align with a master clock of the mobile host terminal, and determine a spatial relative position of the left earphone, the right earphone and the mobile host terminal with a midpoint of the left earphone and the right earphone as an origin.

[0021] A sound source positioning and distance calculation module is configured to receive sound data by the left earphone, the right earphone and the mobile host terminal respectively, calculate a time difference between each two of the three devices, convert the time difference into a distance difference Δd based on a current sound speed, rl Δd lp Δd rp and substitute the distance difference Δd into a hyperbolic equation set to solve a three-dimensional (X, Y, Z) coordinate of a sound source by a nonlinear optimization algorithm, and determine distance information of the sound source and the mobile host terminal based on the coordinate value.

[0022] A semantic analysis and priority value determination module is configured to perform semantic analysis of a current scene based on the sound data and the distance information, and determine an interrupt priority value based on a semantic analysis result, the interrupt priority value representing an importance degree of the corresponding sound data.

[0023] An interrupt reminding module is configured to issue a reminder to a user based on sound data with the highest interrupt priority value and greater than a preset threshold.

[0024] Preferably, the device connection and positioning module is specifically configured to establish a Cartesian coordinate system with the midpoint of the left earphone and the right earphone as an origin, with an earphone baseline direction as an x-axis, a horizontal direction perpendicular to the x-axis as a y-axis, and a vertical direction perpendicular to the x-axis and the y-axis as a z-axis, and determine a relative coordinate (Xp, Yp, Zp) of the mobile host terminal by a sensor, with an initial coordinate of the left earphone being (-0.09, 0, 0) and an initial coordinate of the right earphone being (0.09, 0, 0).

[0025] Preferably, the sensor used in the device connection and positioning module is a Bluetooth sensor or a UWB sensor, and specifically comprises:

[0026] A detection unit is configured to send detection data packets to the left earphone and the right earphone respectively when detecting that a spatial position of the mobile host terminal is relatively stable.

[0027] The ranging unit is configured to return a response data packet immediately after the left earphone and the right earphone receive the probe data packet, and the mobile host end records the round trip time tl, tr and the data processing time tc of the earphone from the response data packet after receiving the response data packet; the distance d between the mobile host end and the left earphone is calculated lp =(tl-tc)×c / 2, wherein c is the speed of light; the distance d between the mobile host end and the right earphone is calculated rp =(tr-tc)×c / 2, wherein c is the speed of light;

[0028] The solving unit is configured to establish a left spherical equation through the initial coordinates of the left earphone, d lp establish a right spherical equation through the initial coordinates of the right earphone, d rp establish a right spherical equation through the initial coordinates of the right earphone, d

[0029] The coordinate determination unit is configured to determine the final spatial position relative coordinates (Xp, Yp, Zp) of the mobile host end in combination with the preset user mobile host end habit position and the circular relative position range.

[0030] Preferably, the sensor used in the device connection and positioning module is an IMU sensor, and specifically includes:

[0031] The IMU initialization unit is configured to assume the initial position (0, -1, 0) of the mobile host end and set a constraint condition;

[0032] The update optimization unit is configured to keep the spatial positions of the left earphone, the right earphone and the mobile host end updated according to the IMU measurement values of the three, and optimize the spatial positions of the left earphone, the right earphone and the mobile host end according to the constraint condition;

[0033] The convergence judgment unit is configured to judge whether the spatial positions of the three converge to a specific spatial range, and if so, output the spatial positions of the three, and if not, trigger the update optimization unit to execute cyclically.

[0034] Preferably, the semantic analysis and priority value determination module specifically includes:

[0035] The first recognition unit is configured to determine the spectral features of different sounds from the sound spectrum graph, and train a sound data classification recognition model of different sound data using a sound spectrum feature set;

[0036] The second recognition unit is configured to determine the sound type according to the current sound data and the trained sound data classification recognition model, and if the sound type corresponds to a specific meaning, calculate the interrupt priority value in combination with the distance information; if the sound type is a voice event, convert it into text for semantic recognition, and calculate the interrupt priority value according to the semantic recognition result and the distance information.

[0037] The application provides an implementation method of an intelligent earphone interrupt function based on positioning technology, at least achieves the following technical effects: (1) through multi-device clock synchronization and space positioning technology, the precise measurement of the three-dimensional position of a sound source is realized, the spatial precision of environmental sound perception is effectively improved; three sensor device space positioning algorithms are given, which are different from the prior art route and stereoscopic layout of intelligent earphone interaction technology; (2) through the fusion of sound semantic recognition and sound source distance information for interrupt priority value calculation, the system can intelligently judge the importance of sound events, and trigger an interrupt reminder only in a truly critical scene, which avoids invalid disturbance and ensures that important information is not missed; (3) the whole scheme realizes the leap from passive listening to active environmental perception and intelligent decision-making, significantly improving the interaction intelligence and user experience of the intelligent earphone in complex scenes. BRIEF DESCRIPTION OF DRAWINGS

[0038] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings.

[0039] Figure 1 is a flowchart of an implementation method of an intelligent earphone interrupt function based on positioning technology according to an embodiment of the application.

[0040] Figure 2 is a structural block diagram of an implementation system of an intelligent earphone interrupt function based on positioning technology according to an embodiment of the application. DETAILED DESCRIPTION

[0041] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0042] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0044] Figure 1 The diagram illustrates the four main steps of the core method of this invention: device connection and relative positioning (S100), sound source localization and distance calculation (S110), semantic analysis and priority value determination (S120), and interruption reminder (S130). The diagram clearly depicts the complete process from system initialization and sound source spatial localization to intelligent interruption decision-making.

[0045] Figure 2 Showing with Figure 1 The system architecture corresponding to the method includes four core modules: the first positioning module, the second positioning module, the semantic analysis and priority value determination module, and the interruption reminder module. This block diagram reveals the connections and data interaction relationships between the various modules within the system.

[0046] Example 1: The first aspect is as follows Figure 1 As shown, the present invention discloses an embodiment of a method for implementing the interruption function of a smart earphone based on positioning technology, including:

[0047] 1. A method and system for implementing an interruption function in smart headphones based on positioning technology, the method comprising the following steps:

[0048] S100 establishes a connection between the left and right earpieces and the mobile host through a connection protocol. The left and right earpieces adjust their clock offsets to align with the main clock of the mobile host. The spatial relative positions of the left and right earpieces and the mobile host are determined with the midpoint of the left and right earpieces as the origin.

[0049] Only three devices can accurately locate the sound source by explicitly positioning their own space, which is the basis and premise of the whole scheme. Before positioning, first of all, through the wireless connection protocol such as Bluetooth connection, the connection between the left and right earphones and the mobile host is established, and the internal clock of the two earphones is aligned with the "master clock" of the mobile host, eliminating the timestamp error between the three.

[0050] The mobile host is, for example, a mobile phone, earphone charging box, smart watch or other electronic devices carried on the body, which is connected with the Bluetooth earphone as the host and has certain computing power. If a Bluetooth sensor is used, then in the connected state, the master device (such as a mobile phone, smart watch) and the slave device (earphone) will adjust their respective clocks through a mechanism called "clock drift" to keep them synchronized and ensure accurate communication timing, providing a unified and high-precision time reference system for subsequent sensor data. If the clocks of the two devices are not synchronized, their timestamps are not comparable. In a preferred embodiment, the timestamp based on the clock offset is marked on the time axis as the reference time node, and the local timestamp is marked on the time axis as the time node to be aligned. Linear interpolation is performed based on the local data at the reference time node to generate local data corresponding to the reference time node. The earphone data corresponding to the reference time node is mapped to the local data corresponding to the reference time node to obtain a mapping data list, thereby completing the alignment operation. Clock alignment is important for signal processing between multiple devices. Usually, a Bluetooth sensor can be used as one of the multi-mode sensors to perform clock alignment, but it can also cooperate with other sensors or use other more optimal clock alignment operations.

[0051] In one embodiment, a Cartesian coordinate system is established with the midpoint of the left earphone and the right earphone as the origin, the earphone baseline direction as the x-axis, the horizontal and vertical direction perpendicular to the x-axis as the y-axis, and the z-axis perpendicular to the x and y axes. In one embodiment, since the displacement of the earphone relative to the head can be ignored, the default value of the earphone coordinate is directly set, i.e. the initial coordinate of the left earphone is (-0.09, 0, 0) and the coordinate of the right earphone is (0.09, 0, 0). Further, the relative coordinates (Xp, Yp, Zp) of the mobile host are determined by the sensor.

[0052] In a preferred embodiment, the sensor is a Bluetooth sensor. The method is as follows: (1) under the condition that the spatial position of the mobile host is relatively stable, probe data packets are sent to the left earphone and the right earphone respectively; (2) the left earphone and the right earphone return reply data packets immediately after receiving the probe data packets, and the mobile host records the round-trip time tl, tr and the data processing time tc of the earphone obtained from the reply data packet after receiving the reply data packet; (3) the distance d lp= (tl-tc) x c / 2, where c is the speed of light; the distance d between the mobile host and the right earphone rp = (tr-tc) x c / 2, where c is the speed of light; the initial coordinate of the left earphone, d lp establish the left spherical equation, through the initial coordinate of the right earphone, d rp establish the right spherical equation, and solve the circular relative position range of the mobile host by combining the equations; (4) preset the habitual position of the user's mobile host, and determine the final spatial position relative coordinates (Xp, Yp, Zp) of the mobile host by combining the circular relative position range. In the above scheme, whether the mobile host is in a relatively stable state can be determined by the IMU sensor. In the above scheme, the user's mobile host can be within the range of a handheld phone, such as answering a phone (this special case can directly determine the relative position of the phone), the average height of a regular pocket, the range of the human body (below the head, within the shoulder width), etc.

[0053] The use of Bluetooth sensors can achieve the relative coordinates of the mobile host within a certain accuracy / ideal situation, however, in specific experiments and actual applications, the inventors found that the hardware and communication protocols of most existing Bluetooth devices are designed for non-high-precision ranging, and there are also interferences in clock synchronization, noise filtering, and object blocking, etc., resulting in that this spatial positioning method is only suitable for a small number of high-end Bluetooth devices, and the effect in most Bluetooth devices limited by cost does not reach the expected accuracy.

[0054] In another preferred embodiment, the sensor is an IMU sensor, and the method is: (1) assuming an initial position (0, -1, 0) of the mobile host, setting a constraint condition; (2) updating the spatial positions of the left earphone, the right earphone, and the mobile host according to the IMU measurement values of the three; optimizing the spatial positions of the left earphone, the right earphone, and the mobile host according to the constraint condition; (3) judging whether the spatial positions of the three converge in a specific spatial range, and if so, outputting the spatial positions of the three, and if not, executing (2). In the above scheme, after the spatial positions of the three are updated each time by the IMU measurement values, the updated spatial positions need to be optimized again based on the set constraint condition. When the finally obtained spatial position values change in a specific spatial range in a clear regularity or are stable in a specific spatial range, it can be considered that the convergence condition is met. Therefore, the assumed initial position of the left earphone, the right earphone, and the mobile host does not need to strictly conform to the body shape feature or human habit, but the closer it is to the true position, the faster the convergence speed is. In the above scheme, the setting of the constraint condition is the key to determining the accuracy of the relative spatial positions of the left earphone, the right earphone, and the mobile host and achieving fast convergence. In one embodiment, the constraint condition A of the earphone is to match the angular velocity of the head rotation with the rotational linear velocity corresponding to the x-axis coordinate of the earphone, and if the matching fails, the x-axis coordinate of the earphone is adjusted. The constraint condition B of the earphone is to make the x-axis coordinates of the updated positions of the left earphone and the right earphone symmetrical based on the origin, otherwise, the x-axis coordinates are adjusted in the symmetrical direction. The constraint condition C of the earphone is to stabilize the motion state in the baseline and the x-axis level. In one embodiment, the constraint condition A of the mobile host is a physical feasible domain constraint, including defining a body model, and if the motion trajectory collides with the body model, a penalty term is set to indicate coordinate adjustment. The constraint condition B is a kinematics constraint, and if the detected motion state violates the conventional motion law, a penalty term is set to adjust the coordinate based on the principle of smooth motion. The constraint condition C is a probability domain constraint based on activity, which is applied to a neural network model: first, a motion recognition model is established based on historical IMU data, second, the current IMU data is classified based on the motion recognition model, the classification result with a confidence higher than a preset threshold is taken as a reliable recognition result, and the reliable recognition result is converted into a priori coordinate value, the priori loss of the priori coordinate value and the coordinate value updated based on the current IMU data is judged, and the motion state is adjusted based on the priori loss to obtain the updated coordinate value of the mobile host. In the above scheme, the method for judging whether the spatial positions of the three converge in a specific spatial range is: (1) filtering the same motion component; (2) calculating the ratio of the mean value and the standard deviation of the device coordinate sequence, and if the ratio is less than a preset value (such as 5%), it is considered that the device spatial position converges in a specific spatial range.

[0055] The accuracy of determining the spatial relative positions of the left earphone, the right earphone and the mobile host by using the IMU sensor is obviously higher than that of using the Bluetooth sensor, and the requirement for the performance of the equipment is not particularly strict. At present, relatively advanced earphones on the market are all equipped with relevant IMU sensors, and therefore the hardware adaptability is relatively good. However, this method has a relatively high requirement for the algorithm setting, and in actual application, the algorithm constraint should be continuously verified, analyzed and adjusted, and a more accurate model should be trained to improve the positioning accuracy of the relative positions of the three devices.

[0056] In another preferred embodiment, the sensor is a UWB sensor. The method is as follows: (1) under the condition that the spatial position of the mobile host is relatively stable, sending a probe data packet to the left earphone and the right earphone respectively; (2) the left earphone and the right earphone return a response data packet immediately after receiving the probe data packet, and the mobile host records the round-trip time tl, tr and the data processing time tc of the earphone obtained from the response data packet; (3) the distance d lp =(tl-tc)×c / 2, where c is the speed of light; the distance d rp =(tr-tc)×c / 2, where c is the speed of light; the initial coordinates of the left earphone, d lp establishing a left spherical equation, the initial coordinates of the right earphone, d rp establishing a right spherical equation, and solving the relative circular position range of the mobile host by combining the equations; (4) presetting the habitual position of the user's mobile host, and determining the final spatial position relative coordinates (Xp, Yp, Zp) of the mobile host by combining the relative circular position range. In the above scheme, whether the mobile host is in a relatively stable state can be determined by using the IMU sensor. In the above scheme, the user's mobile host can be in the range of a handheld phone, such as answering a phone (in this special case, the relative position of the phone can be directly determined), the average height of a regular pocket, the range of a human body (below the head and within the shoulder width), etc.

[0057] The UWB signal can significantly improve the ranging capability compared with the Bluetooth signal due to the extremely short pulse, and has strong anti-interference ability and high refresh rate. In the test, the accuracy can reach the centimeter level, and the overall reliability is high, which meets the actual application requirements. However, the price and ecology are relatively inferior to the Bluetooth, and at present only a few high-end intelligent earphones are equipped with UWB sensors, but from the future, using UWB to replace Bluetooth is a trend, and it is suitable to lay out in advance.

[0058] In S110, the left earphone, the right earphone and the mobile host receive sound data respectively, calculate the time difference between the three devices, and convert the time difference into a distance difference based on the current speed of sound Δd rl , Δd lp , Δd rpAnd substitute into the hyperbolic equation group, through the nonlinear optimization algorithm to solve the three-dimensional (X, Y, Z) coordinates of the sound source, based on the coordinate value to determine the distance information between the sound source and the mobile host;

[0059] All the distance difference between two fixed points (A and B) is a constant value, the set of points, consisting of a rotating hyperboloid, the sound source must be located on this hyperboloid, by setting three sound receiving devices, forming three hyperbolic equation, the above three hyperbolic direction simultaneous equations can be solved to accurately determine the coordinate position of the sound source in space, solve the problem of dimension missing and inaccurate positioning in the prior art. In the preferred embodiment, the optimal solution is found by least square method, Newton iteration method and other nonlinear optimization algorithms, that is, the sound source coordinates that best satisfy the three time difference equations. Finally, based on the sound source coordinate value, the distance information between the sound source and the origin is determined.

[0060] In a preferred embodiment, there may be multiple optimal solutions, which need to be excluded by applying environmental constraints, for example, the coordinate value below the ground should be excluded; in special cases, the left earphone, right earphone and mobile host form a plane perpendicular to the ground, at this time, unreasonable coordinate values cannot be simply excluded, but there are still some solutions. In one embodiment, a frequency spectrum clue is added, the sound data collected by the earphone microphone is sent to the mobile host, the sound spectrum is analyzed, if the high frequency is missing, the front coordinate is excluded, otherwise the rear coordinate is excluded; in another embodiment, a Doppler frequency shift and phase change motion clue is added, if a sharp sound (such as a whistle) is detected, its pitch is continuously and stably rising, which can almost be determined that a moving sound source is approaching from the rear; on the contrary, if the pitch is falling, it may be that the sound source is moving away or coming from the front, so the coordinates can be excluded according to the motion clue. In a more preferred embodiment, multi-device sound data labeled with true direction is collected for training the orientation recognition model, when new sound data arrives, the sound data is input into the orientation recognition model to determine the most probable orientation of the sound source, so as to exclude the remaining orientations.

[0061] S120, based on the sound data and the distance information, performing semantic analysis of the current scene, determining a interrupt priority based on the semantic analysis result, the interrupt priority representing an importance degree of the corresponding sound data.

[0062] The semantic analysis is not only speech recognition ASR, but at least includes scene recognition, event recognition, speech keyword recognition. When in a noisy environment with multiple voices, the system calculates a comprehensive "interrupt priority value" for one or more simultaneous sound events. Only when this value exceeds a certain preset threshold, the system will remind the user through the earphone (for example, play a prompt tone, broadcast the content "vehicle approaching behind"), otherwise it will remain silent and not disturb the user, and before issuing a reminder, the original sound of the earphone is reduced in volume or stopped playing, thereby highlighting the reminder information.

[0063] The importance of the same sound at different distances is completely different. In the prior art, it is usually directly analyzed whether the sound emitted by the current scene contains dangerous factors, such as whether a gunshot is recognized, a car horn is close, so as to determine whether to prompt the user; the present application uses distance information as a link of semantic analysis for complex scenes, improves the richness of information, and more accurately expresses what is happening in the current scene. This interaction is more intelligent and more in line with user needs.

[0064] In a preferred embodiment, first, the spectral features of different sounds are determined from the sound spectrum graph, and a sound data classification recognition model is trained for different sound data; second, the sound type is determined according to the current sound data and the trained sound data classification recognition model, and if the sound type corresponds to a specific meaning, the interrupt priority value is calculated in combination with the distance information; if the sound type is a voice event, it is converted into text for semantic recognition, and the interrupt priority value is calculated according to the semantic recognition result and the distance information.

[0065] The interrupt priority value is a rule engine or a machine learning model. In the case of rule-based, it has the advantages of intuitive and strong interpretability, for example, in the case of the same event at different distances, the interrupt priority value is high at close distance, and in the case of the same distance and different events, the interrupt priority value is high for dangerous events. In a preferred embodiment, the calculation of the interrupt priority value can be based on a machine learning method, which models the decision problem as a gradient boosting tree model, and trains the model based on event type, semantic analysis result, event distance, direction confidence, and scene context. The model builds the relationship between the multi-dimensional information such as distance and event type and the interrupt priority value through training samples. At the same time, especially in a noisy environment with multiple voices, there will be multiple possible reminder events. The present application determines the most important and urgent event to prompt the user based on the interrupt priority value.

[0066] S130, issuing a reminder to the user based on the sound data with the highest interrupt priority value and greater than a preset threshold.

[0067] In a preferred embodiment, the prompt content is directly conveyed for the sound data, for example, when a passerby greets the person wearing the earphone, the earphone sends a reminder to the user, and the reminder content is the greeting content of the passerby "Hello, how do I get to the park?"; in a preferred embodiment, the prompt content is a translation of the semantic analysis result, for example, when a passerby greets the person wearing the earphone, the earphone sends a reminder to the user, and the reminder content is "Master, 3 meters to the left rear, someone asks you for directions" "Master, 10 meters to the right front, there may be a crowd running quickly, please be careful"; in a preferred embodiment, the prompt content changes correspondingly based on the different interrupt priority values, for example, when the situation is urgent, the interrupt priority value is higher, at this time, the sound can be adjusted to be higher, and the content can be simplified to find a quick and clear prompt expression, for example, in an emergency, a short, flat and fast way can be used to repeatedly summarize "8 meters to the right, be careful"; at the same time, the intelligent earphone continues to maintain the interactive state with the user, and when receiving an instruction such as the user's desire to obtain detailed information, the corresponding information is fed back to the user. For sound data with an interrupt priority value less than a preset threshold, it is unnecessary to feed back to the user; for sound data with an interrupt priority value greater than the preset threshold, the user can be fed back in turn.

[0068] Embodiment two: the second aspect is as shown in the description Figure 2 The present disclosure further provides an implementation system of the interrupt function of the intelligent earphone based on the positioning technology, which comprises:

[0069] The device connection and positioning module is configured to establish the connection of the left intelligent earphone, the right intelligent earphone and the mobile host terminal through a connection protocol, to adjust the clock offset of the left earphone and the right earphone to align with the master clock of the mobile host terminal, and to determine the spatial relative positions of the left earphone, the right earphone and the mobile host terminal with the midpoint of the left earphone and the right earphone as the origin.

[0070] The sound source positioning and distance calculation module is configured to receive sound data by the left earphone, the right earphone and the mobile host terminal respectively, to calculate the time difference between each two of the three devices, to convert the time difference into a distance difference Δd rl , Δd lp , Δd rp , and to substitute the distance difference into the hyperbolic equation set to solve the three-dimensional (X, Y, Z) coordinates of the sound source by a nonlinear optimization algorithm, and to determine the distance information of the sound source and the mobile host terminal based on the coordinate values.

[0071] The semantic analysis and priority value determination module is configured to perform semantic analysis of the current scene based on the sound data and the distance information, and to determine the interrupt priority value based on the semantic analysis result, wherein the interrupt priority value represents the importance of the corresponding sound data.

[0072] The interrupt reminding module is configured to remind the user based on the sound data with the highest interrupt priority value and greater than a preset threshold.

[0073] In a preferred embodiment, the device connection and positioning module is specifically configured to: establish a Cartesian coordinate system with the midpoint of the left earphone and the right earphone as the origin, the earphone baseline direction as the x-axis, the horizontal and vertical direction perpendicular to the x-axis as the y-axis, and the z-axis perpendicular to the x-axis and the y-axis; the initial coordinates of the left earphone are (-0.09, 0, 0), and the initial coordinates of the right earphone are (0.09, 0, 0); and the relative coordinates (Xp, Yp, Zp) of the mobile host end are determined by the sensor.

[0074] In a preferred embodiment, the sensor used in the device connection and positioning module is a Bluetooth sensor or a UWB sensor, and specifically includes:

[0075] The detection unit is configured to send detection data packets to the left earphone and the right earphone respectively when it is detected that the spatial position of the mobile host end is relatively stable;

[0076] The ranging unit is configured to return response data packets immediately after the left earphone and the right earphone receive the detection data packets, and the mobile host end records the round-trip time tl, tr and the data processing time tc of the earphone from the response data packets after receiving the response data packets; the distance d lp between the mobile host end and the left earphone is calculated as (tl-tc)×c / 2, where c is the speed of light; and the distance d rp between the mobile host end and the right earphone is calculated as (tr-tc)×c / 2, where c is the speed of light.

[0077] The solving unit is configured to establish a left spherical equation through the initial coordinates of the left earphone and d lp , and establish a right spherical equation through the initial coordinates of the right earphone and d rp , and solve the circular relative position range of the mobile host end by combining the equations.

[0078] The coordinate determination unit is configured to determine the final spatial position relative coordinates (Xp, Yp, Zp) of the mobile host end by combining the preset habit position of the user's mobile host end and the circular relative position range.

[0079] In a preferred embodiment, the sensor used in the device connection and positioning module is an IMU sensor, and specifically includes:

[0080] The IMU initialization unit is configured to assume the initial position of the mobile host end as (0, -1, 0) and set a constraint condition.

[0081] The update optimization unit is configured to update the spatial positions of the left earphone, the right earphone and the mobile host according to IMU measurement values of the left earphone, the right earphone and the mobile host, and optimize the spatial positions of the left earphone, the right earphone and the mobile host according to constraint conditions.

[0082] The convergence judgment unit is configured to judge whether the spatial positions of the three converge in a specific spatial range, and if yes, output the spatial positions of the three, and if not, trigger the update optimization unit to perform cyclically.

[0083] In a preferred embodiment, the semantic analysis and priority value determination module specifically comprises:

[0084] The first recognition unit is configured to determine spectral features of different sounds from the sound spectrum graph, and train a sound data classification and recognition model of different sound data by using a sound spectrum feature set;

[0085] The second recognition unit is configured to determine a sound type according to current sound data and the trained sound data classification and recognition model, calculate a break priority value in combination with the distance information if the sound type corresponds to a specific meaning, convert the sound type into text for semantic recognition if the sound type is a voice event, and calculate a break priority value according to the semantic recognition result and the distance information.

[0086] The application provides an implementation method of an intelligent earphone break function based on positioning technology, and at least achieves the following technical effects: (1) through multi-device clock synchronization and spatial positioning technology, the precise measurement of the three-dimensional position of a sound source is realized, and the spatial precision of environmental sound perception is effectively improved; three spatial positioning algorithms of sensor devices are given, which are different from the prior art route and stereoscopic layout of intelligent earphone interaction technology; (2) through the fusion of sound semantic recognition and sound source distance information for break priority value calculation, the system can intelligently judge the importance of a sound event, and trigger a break reminder only in a truly critical scene, thereby avoiding invalid disturbance and ensuring that important information is not missed; (3) the whole scheme realizes the leap from passive listening to active environmental perception and intelligent decision-making, and significantly improves the interaction intelligence and user experience of the intelligent earphone in a complex scene.

[0087] According to an embodiment, a program product, such as a machine-readable medium, is provided. The machine-readable medium can have instructions (i.e., the above-mentioned elements implemented in software) that, when executed by a machine, cause the machine to perform various operations and functions described in this specification. Figure 1 Specifically, a system or apparatus equipped with a readable storage medium on which a software program code implementing the functions of any of the above-mentioned embodiments is stored, and a computer or processor of the system or apparatus reading and executing the instructions stored in the readable storage medium can be provided.

[0088] In this case, the program code itself read from the readable medium can implement the functions of any of the above-described embodiments, and thus the machine-readable code and the readable storage medium storing the machine-readable code form part of the present specification.

[0089] Embodiments of the readable storage medium include a floppy disk, a hard disk, a magneto-optical disk, an optical disk such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW, a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer or a cloud over a communication network.

[0090] Those skilled in the art should understand that the various embodiments disclosed above can be variously changed and modified without departing from the essence of the present application. Therefore, the scope of protection of the present specification should be defined by the appended claims rather than by the detailed description.

[0091] It should be noted that not all steps and units in the above-described flowcharts and system block diagrams are necessary, and some steps or units can be omitted according to actual needs. The execution order of the steps is not fixed and can be determined as needed. The device structure described in the above embodiments can be a physical structure or a logical structure, i.e., some units can be implemented by the same physical client, or some units can be implemented by multiple physical clients, or some units can be implemented by some components in multiple independent devices.

[0092] In the above embodiments, a hardware unit or module can be implemented by mechanical or electrical means. For example, a hardware unit, module or processor can include permanent dedicated circuitry or logic (such as a dedicated processor, FPGA or ASIC) to perform the corresponding operation. The hardware unit or processor can also include programmable logic or circuitry (such as a general-purpose processor or other programmable processor), which can be temporarily set by software to perform the corresponding operation. The specific implementation method (mechanical method, or dedicated permanent circuit, or temporarily set circuit) can be determined based on cost and time considerations.

[0093] The detailed description set forth above describes exemplary embodiments but is not intended to represent the only embodiments in which the claimant can realize and use the present disclosure. The term "exemplary" is used herein to mean "serving as an example, instance, or illustration," and not to mean "preferred" or "having some other perceived advantage." The detailed description includes specific details for the purpose of providing a thorough understanding of the technology described. However, it will be apparent to those skilled in the art that the technology described can be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form in order to avoid obscuring the concepts of the described embodiments.

[0094] The foregoing description of the present disclosure has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the disclosure be limited not with this detailed description, but rather by the claims appended hereto.

Claims

1. An implementation method of an intelligent earphone interrupt function based on positioning technology, the method comprising the steps of: S100, establishing a connection between a left intelligent earphone, a right intelligent earphone and a mobile host terminal through a connection protocol, adjusting a clock offset of the left earphone and the right earphone to align with a master clock of the mobile host terminal, and determining spatial relative positions of the left earphone, the right earphone and the mobile host terminal with a midpoint of the left earphone and the right earphone as an origin; S110, the left earphone, the right earphone and the mobile host respectively receive sound data, calculate the time difference between each two of the three devices, and convert it into distance difference Ad based on the current sound speed rl , Ad lp , Ad rp , and substitute into the hyperbolic equation group, solve the three-dimensional (X, Y, Z) coordinates of the sound source through a nonlinear optimization algorithm, and determine the distance information between the sound source and the mobile host based on the coordinate value; S120, performing semantic analysis of a current scene based on the sound data and the distance information, and determining an interrupt priority value based on a semantic analysis result, the interrupt priority value representing an importance degree of corresponding sound data; S130, issuing a reminder to a user based on sound data with the highest interrupt priority value and greater than a preset threshold value; wherein the determination of the spatial relative positions of the left earphone, the right earphone and the mobile host terminal with the midpoint of the left earphone and the right earphone as the origin comprises: establishing a Cartesian coordinate system with the midpoint of the left earphone and the right earphone as an origin, an earphone baseline direction as an x-axis, a horizontal direction perpendicular to the x-axis as a y-axis, and a vertical direction perpendicular to the x-axis and the y-axis as a z-axis; an initial coordinate of the left earphone is (-0.09, 0, 0), and an initial coordinate of the right earphone is (0.09, 0, 0); and a relative coordinate (Xp, Yp, Zp) of the mobile host terminal is determined through a sensor; The sensor is a Bluetooth sensor or a UWB sensor, and the determination of the relative coordinates (Xp, Yp, Zp) of the mobile host end by the sensor includes the following steps: (1) when it is detected that the spatial position of the mobile host end is relatively stable, a probe data packet is sent to the left earphone and the right earphone respectively; (2) the left earphone and the right earphone return a response data packet immediately after receiving the probe data packet, and the mobile host end records the round trip time tl and tr and the data processing time tc of the earphone obtained from the response data packet after receiving the response data packet; (3) the distance d lp =(tl-tc)×c / 2, where c is the speed of light; the distance d rp =(tr-tc)×c / 2, where c is the speed of light; the initial coordinates of the left earphone, d lp a left spherical equation is established, the initial coordinates of the right earphone, d rp a right spherical equation is established, and the circular relative position range of the mobile host end is obtained by solving the equations; and (4) the habitual position of the user's mobile host end is preset, and the final spatial position relative coordinates (Xp, Yp, Zp) of the mobile host end are determined in combination with the circular relative position range.

2. The method of claim 1, wherein: the sensor is an IMU sensor, and the method comprises the steps of: (1) assuming an initial position (0, -1, 0) of the mobile host terminal and setting a constraint condition; (2) updating spatial positions of the left earphone, the right earphone and the mobile host terminal according to IMU measurement values of the three; and optimizing the spatial positions of the left earphone, the right earphone and the mobile host terminal according to the constraint condition; and (3) judging whether the spatial positions of the three converge in a specific spatial range, and if yes, outputting the spatial positions of the three, and if not, cyclically executing (2).

3. The method of claim 1, wherein: First, spectral features of different sounds are determined from a sound spectrum graph, and a sound data classification and recognition model of different sound data is trained based on a sound feature set; second, a sound type is determined according to current sound data and the trained sound data classification and recognition model, and if the sound type corresponds to a specific meaning, an interrupt priority value is calculated in combination with the distance information; and if the sound type is a speech event, the speech event is converted into text for semantic recognition, and an interrupt priority value is calculated according to a semantic recognition result and the distance information.

4. A system for implementing an intelligent earpiece interrupt function based on a positioning technology, characterized in that, The system comprises: a first positioning module configured to establish a connection between a left intelligent earphone, a right intelligent earphone and a mobile host terminal through a connection protocol, adjust a clock offset of the left earphone and the right earphone to align with a master clock of the mobile host terminal, and determine spatial relative positions of the left earphone, the right earphone and the mobile host terminal with a midpoint of the left earphone and the right earphone as an origin; The second positioning module is configured to receive sound data through the left earphone, the right earphone and the mobile host respectively, calculate time differences between the three devices in pairs, and convert the time differences into distance differences Δd based on a current sound speed rl , Δd lp , Δd rp , and substitute into a hyperbolic equation set, and solve three-dimensional (X, Y, Z) coordinates of the sound source through a nonlinear optimization algorithm to determine distance information of the sound source and the mobile host based on the coordinate values. a semantic analysis and decision module configured to perform semantic analysis of a current scene based on the sound data and the distance information, and determine an interrupt priority value based on a semantic analysis result, the interrupt priority value representing an importance degree of corresponding sound data; an interrupt reminder module configured to issue a reminder to a user based on sound data with the highest interrupt priority value and greater than a preset threshold value. The first positioning module is further configured to: establish a Cartesian coordinate system with the midpoint of the left earphone and the right earphone as the origin, the earphone baseline direction as the x-axis, the horizontal and vertical direction perpendicular to the x-axis as the y-axis, and the direction perpendicular to the x-axis and the y-axis as the z-axis; the initial coordinates of the left earphone are (-0.09, 0, 0), and the coordinates of the right earphone are (0.09, 0, 0); and the relative coordinates (Xp, Yp, Zp) of the mobile host are determined by the sensor. The first positioning module uses a Bluetooth sensor or a UWB sensor, including: a detection unit configured to send detection data packets to the left and right earpieces respectively when the spatial position of the mobile host is detected to be relatively stable; a ranging unit configured to immediately return response data packets after the left and right earpieces receive the detection data packets, and the mobile host records the round-trip time tl, tr, and the data processing time tc for parsing the earpiece data from the response data packets after receiving the response data packets; and calculates the distance d between the mobile host and the left earpiece. lp =(tl-tc)×c / 2, where c is the speed of light; calculate the distance d between the mobile host and the right earphone. rp =(tr-tc)×c / 2, where c is the speed of light; the solution unit is configured to use the initial coordinates of the left earphone and d. lp Establish the equation of the left sphere, using the initial coordinates of the right earpiece and d. rp Establish the equation of the right sphere, and solve the equation simultaneously to obtain the circular relative position range of the mobile host terminal; the coordinate determination unit is configured to combine the preset user mobile host terminal's habitual position with the circular relative position range to determine the final spatial position relative coordinates (Xp, Yp, Zp) of the mobile host terminal.

5. The system of claim 4, wherein, The sensor used in the first positioning module is an IMU sensor, and specifically includes: an IMU initialization unit configured to assume the initial position of the mobile host as (0, -1, 0) and set a constraint condition; an update optimization unit configured to maintain the update state of the spatial positions of the left earphone, the right earphone, and the mobile host according to the IMU measurement values of the three, and optimize the spatial positions of the left earphone, the right earphone, and the mobile host according to the constraint condition; and a convergence judgment unit configured to judge whether the spatial positions of the three converge to a specific spatial range, and if so, output the spatial positions of the three, and if not, trigger the update optimization unit to execute in a loop.

6. The system of claim 4, wherein, The semantic analysis and decision module specifically includes: a first recognition unit configured to determine the spectral features of different sounds from the sound spectrum graph, and train a sound data classification recognition model of different sound data using the sound spectrum feature set; a second recognition unit configured to determine the sound type according to the current sound data and the trained sound data classification recognition model, calculate the interruption priority value in combination with the distance information if the sound type corresponds to a specific meaning, convert the sound type into text for semantic recognition if the sound type is a speech event, and calculate the interruption priority value according to the semantic recognition result and the distance information.

Images