Object tracking method, mobile intelligent device, medium and product

Abstract

The invention discloses an object tracking method, a mobile intelligent device, a medium and a product, and relates to the mobile intelligent device. The method comprises the following steps: when a user object walks in a target area, receiving an area sound signal collected by the acoustic sensor; extracting a target foot sound signal of the user object from the area sound signal; determining relative position information and walking speed of the user object according to the target footstep sound signal; and controlling the moving direction and the moving speed of the mobile device to be matched with the user object according to the relative position information and the walking speed so as to track the display device. The footstep sound signals of the user are accurately captured and tracked through the acoustic sensor, real-time monitoring of user actions and automatic following of the mobile device are achieved, and interactivity and user experience of the device anytime and anywhere are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of mobile intelligent devices, and in particular to an object tracking method, a mobile intelligent device, a medium and a product. Background Art

[0002] Mobile smart devices include portable smart devices such as girlfriend phones and mobile smart screens. They can connect to the Internet through wireless networks, have certain computing capabilities, and are portable electronic devices that can run applications.

[0003] Current mobile smart devices are equipped with universal wheels and other moving devices at the bottom, so that they can be pushed by users and provide companion services and information interaction for consumers anytime and anywhere. This design allows mobile smart devices to be easily moved between different scenes in the home, such as from the living room to the kitchen, study, bedroom and even bathroom, to meet the needs of users in different occasions. They can also be used in offices, exhibition halls, hotels, airports and other scenes.

[0004] However, most of the current mobile smart devices require users to expend a certain amount of effort to push them before they can move. They cannot automatically follow the user's movements and lack flexibility and convenience in use. Summary of the invention

[0005] In view of the above technical problems and defects, the purpose of the present invention is to provide an object tracking method, a mobile smart device, a medium and a product, which solve the technical problem that the mobile smart device cannot automatically follow the user's movement trajectory.

[0006] To achieve the above-mentioned purpose, in a first aspect, the present invention provides an object tracking method, which is applied to a display device of a mobile smart device, wherein the mobile smart device also includes a mobile device and an acoustic sensor, the display device is arranged on the mobile device, and the acoustic sensor is connected to the display device, the method comprising: when a user object walks in a target area, receiving a regional sound signal collected by the acoustic sensor; extracting a target footstep sound signal of the user object from the regional sound signal; determining the relative position information and walking speed of the user object according to the target footstep sound signal; and controlling the moving direction and moving speed of the mobile device to match those of the user object according to the relative position information and the walking speed, so as to track the display device.

[0007] The present invention uses an acoustic sensor to collect regional sound signals and extract the user's footsteps signal from them, which enables the display device to be unrestricted by light conditions and accurately identify and track users even in dimly lit environments or environments with many visual obstacles. Secondly, by analyzing the footsteps signal to determine the user's relative position and walking speed, the display device can adjust the moving direction and speed of the mobile device in real time, keep in sync with the user, and automatically follow the user's movement, enhancing the interactivity and user experience of mobile smart devices anytime and anywhere. In addition, this method also reduces interference with the user's line of sight, and the user does not need to face the camera or other visual tracking devices, thereby improving privacy protection.

[0008] In combination with some embodiments of the first aspect, in some embodiments, the step of extracting the target foot sound signal of the user object from the area sound signal includes: upon confirming the footstep sound characteristics of the user object, extracting the target foot sound signal from the area sound signal according to the footstep sound characteristics.

[0009] The technical solution of the above embodiment is adopted to improve the accuracy and reliability of object tracking. By confirming the characteristics of the footsteps of the user before walking, and extracting the target footsteps sound signal from the regional sound signal based on these characteristics, the footsteps of the user can be distinguished and identified more accurately, especially when multiple people are present at the same time or the ambient noise is large. This method reduces the possibility of misidentification and ensures that the tracking only responds to the footsteps of the target user, thereby improving the accuracy of tracking and the quality of the user interaction experience.

[0010] In combination with some embodiments of the first aspect, in some embodiments, before the step of receiving the area sound signal collected by the acoustic sensor when the user object walks in the area, it also includes: in response to the follow-up instruction of the user object, issuing a pre-movement prompt to the user object; when the user object walks in the target area according to the pre-movement prompt, obtaining a pre-movement sound signal; performing footstep signal recognition on the pre-movement sound signal to obtain the footstep characteristics of the user object.

[0011] By adopting the technical solution of the above embodiment, the interactivity and predictive ability of the mobile smart device are enhanced. By issuing a pre-movement prompt before the user starts to walk, not only can the tracking be prepared in advance, but also the footstep characteristics of the user can be identified by obtaining the pre-movement sound signal. This method allows the user's actions to be anticipated before the user actually moves, thereby adjusting the tracking strategy earlier, reducing response delays, and providing users with a smoother and more timely tracking experience.

[0012] In combination with some embodiments of the first aspect, in some embodiments, before the step of obtaining the pre-walking sound signal, it also includes: determining whether there are other objects within the surrounding range of the user object; if so, prompting the user object to maintain a set distance from the other objects.

[0013] The technical solution of the above embodiment is adopted to optimize the tracking performance in a multi-user environment. By judging whether there are other objects around the user and prompting the user to keep a certain distance when there are other objects, the interference of the footsteps of other objects on the footsteps signal of the target user can be reduced. This helps to maintain the clarity and accuracy of tracking, especially in a crowded environment or an environment where multiple users are moving at the same time, ensuring that the footsteps of different users can be effectively distinguished to avoid tracking errors.

[0014] In combination with some embodiments of the first aspect, in some embodiments, the mobile smart device also includes a camera, and the method also includes: obtaining a character video image of the user object when walking through the camera; predicting an estimated movement direction of the user object based on the character video image; and controlling the movement direction of the mobile device based on the estimated movement direction.

[0015] The technical solution of the above embodiment is adopted to improve the multimodal perception capability of tracking. By acquiring video images of the user walking by integrating the camera and predicting the user's direction of movement based on these images, additional data support can be obtained from visual information. This method of combining acoustic and visual information improves the robustness of tracking, and can maintain accurate tracking of user actions even when the sound signal is interfered or the visual information is limited, thereby enhancing the user experience.

[0016] In combination with some embodiments of the first aspect, in some embodiments, after the step of extracting the target footstep sound signal of the user object from the sound signal of the area, it also includes: determining the movement path of the user object according to the target footstep sound signal; judging whether the movement direction of the user object is correct according to the movement path and the planned navigation path; if not, issuing a prompt to the user object that the movement direction is wrong.

[0017] By adopting the technical solution of the above embodiment, the navigation assistance function of the system is enhanced. By determining the user's action path and comparing it with the planned navigation path, it is possible to determine whether the user's action direction is correct, and issue a prompt when the user deviates from the planned route. This real-time navigation assistance function not only improves the user's action efficiency, but also enhances the practicality of the system, especially in complex environments or areas that the user is not familiar with, providing users with effective navigation support.

[0018] In combination with some embodiments of the first aspect, in some embodiments, after the step of controlling the moving direction and moving speed of the mobile device to match the user object according to the relative position information and the walking speed to track the display device, it also includes: responding to the user object's release follow-up instruction, confirming the current position information; determining a return route according to the current position information and the position information of the charging station; and controlling the mobile device to move toward the charging station according to the return route.

[0019] The technical solution of the above embodiment is adopted to provide an intelligent autonomous homing mechanism. When the user issues a command to cancel the follow-up, the current position can be automatically confirmed, and the homing route can be determined based on the current position and the location information of the charging station, and the mobile device can be controlled to return to the charging station. This autonomous homing function not only improves the autonomy and user convenience of the device, but also helps the energy management of the device, ensuring that the device can safely return to charge when the power is low, thereby extending the service life of the device.

[0020] In a second aspect, an embodiment of the present invention provides a mobile intelligent device, comprising a display device, a mobile device and an acoustic sensor, wherein the display device is arranged on the mobile device, and the acoustic sensor is connected to the display device; the display device comprises one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions so that the data processing unit executes the method described in the first aspect or the second aspect, and any possible implementation manner in the first aspect or the second aspect.

[0021] In a third aspect, the present invention provides a computer-readable storage medium comprising instructions, which, when executed on a display device of the mobile smart device, causes the display device to execute the method described in the first aspect or the second aspect, and any possible implementation of the first aspect or the second aspect.

[0022] In a fourth aspect, the present invention provides a computer program product comprising instructions, which, when the computer program product is run on the display device of the mobile smart device, enables the display device to execute the method described in the first aspect or the second aspect, and any possible implementation method of the first aspect or the second aspect.

[0023] It can be understood that the mobile smart device provided in the second aspect, the storage medium provided in the third aspect, and the computer program product provided in the fourth aspect are all used to execute the method provided by the present invention. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here.

[0024] One or more technical solutions provided by the present invention have at least the following technical effects or advantages: 1. The present invention improves the accuracy and adaptability of object tracking. By combining the use of acoustic sensors and cameras, the system can extract the footsteps of the target user from the regional sound signal and predict the direction of the user's movement in combination with the video image. This multimodal sensing method enables the tracking system to accurately identify and track users in various environmental conditions, including noisy or poorly lit scenes, reducing the possibility of misidentification and tracking errors, and improving the reliability of tracking and user experience.

[0025] 2. The present invention provides intelligent interaction and navigation assistance functions. The device can not only respond to the user's movements in real time, but also predict the user's action path, compare it with the preset navigation path, and correct the user's action direction in time. This intelligent navigation assistance greatly improves the user's action efficiency and safety in complex or unfamiliar environments, reduces the risk of getting lost, and provides users with real-time, context-based navigation support.

[0026] 3. The present invention also provides the beneficial effects of autonomous homing and energy management. When the user cancels the follow command, the device can automatically confirm the current location and plan the homing route to return to the charging station. This not only improves the autonomy of the device and reduces the management burden of the user, but also optimizes the energy consumption of the device, ensuring that the device can safely return to charge when the power is low, extending the use time and life of the device, and enhancing the practicality and economy of the device. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings herein are incorporated into and constitute a part of the specification, showing embodiments consistent with the present invention, and together with the specification, are used to explain the principles of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings: Figure 1 is a structural schematic diagram of a mobile smart device according to an embodiment of the present invention; Figure 2 is a flow chart of an object tracking method according to an embodiment of the present invention; Figure 3 is a flow chart of another object tracking method according to an embodiment of the present invention; Figure 4 It is a schematic diagram of the architecture of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The terms used in the following embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to be limiting of the present invention. As used in the specification of the present invention, the singular expressions "a", "a", "above", "the" and "this" are intended to also include plural expressions, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present invention refers to any or all possible combinations comprising one or more of the listed items.

[0029] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood as implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, unless otherwise specified, the meaning of "plurality" is two or more.

[0030] It should also be noted that, unless otherwise clearly specified and limited, in the embodiments of the present invention, the terms such as "setting" and "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two components; it can be a wired communication connection or a wireless communication connection. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. The embodiments of the present invention are described in detail below.

[0031] As an emerging mobile smart device, the girlfriend machine or mobile smart screen has brought convenience and pleasure to users in many aspects, such as providing a unified viewing experience in different rooms such as the living room, bedroom, and study, supporting AI somatosensory games and AI fitness, and serving as a home song request station, but its movement method that requires users to push it manually is indeed inconvenient. When using the girlfriend machine, users must expend effort to push it, and cannot enjoy the convenience of automatic following. This limitation becomes particularly obvious when users want to use the girlfriend machine in multiple rooms or locations. Each movement requires the user to personally operate or push it, and it cannot automatically follow the user's actions to meet usage needs anytime and anywhere. Therefore, this feature of the girlfriend machine limits its flexibility and convenience of use to a certain extent, affecting the user experience.

[0032] Therefore, an embodiment of the present invention provides an object tracking method, which is applied to the mobile smart device 100. Figure 1 As shown, the smart device 100 includes a display device 101 , a mobile device 102 , and an acoustic sensor 103 . The display device 101 is arranged on the mobile device 102 , and the acoustic sensor is connected to the display device.

[0033] The display device 101 is one of the core components of the mobile smart device. It is usually a high-resolution screen that can display visual information such as videos, images, and user interfaces, providing users with a rich visual experience. This device may integrate touch screen technology to allow users to interact through touch, and may support high-definition resolution and wide color gamut to provide more delicate and vivid images. At the same time, a processor chip is also set in the display device to perform complex data processing and calculations.

[0034] The mobile device 102 is the carrier of the entire mobile smart device, and the design of the mobile device enables the mobile smart device 100 to move freely. The mobile device 102 is usually equipped with wheels or other moving mechanisms to achieve autonomous movement on different terrains or user-driven movement, so that the display device 101 can be flexibly used in different occasions and environments.

[0035] The acoustic sensor 103 is a component in the mobile smart device for capturing sound signals, which can be a microphone or other types of sound wave receivers. The acoustic sensor 103 can be specifically set on the base of the mobile device, and can better collect the footsteps signal of the user object. By collecting sound signals such as the user's footsteps, it helps the device identify and locate the user's position, as well as monitor the user's movement speed and direction, thereby achieving accurate tracking of the display device. This sensor usually has high sensitivity and good frequency response characteristics to ensure that the required sound signals can be accurately captured in various environments.

[0036] In this embodiment, the acoustic sensor 103 can adopt a microphone array, which is a system composed of multiple microphones that are precisely arranged to capture the spatial information of the sound signal. By analyzing the time difference and intensity difference of the sound reaching each microphone, the microphone array can realize functions such as sound source localization, noise suppression and sound enhancement, so as to accurately extract the target sound in a noisy environment and provide accurate sound signal processing capabilities for the display device of the mobile smart device.

[0037] In some embodiments, the mobile smart device 100 may also include a camera 104 for capturing real-time video images of a user object as it moves in a target area. The camera can provide visual information to assist the acoustic sensor in more accurate user positioning and tracking, while recording the user's path of movement and posture changes. Through the video images captured by the camera, the device can analyze the user's movement pattern, optimize the tracking algorithm, and improve the accuracy and response speed of tracking. In addition, the camera can also be used for interactive functions such as face recognition and gesture recognition to enhance the intelligent interactive capabilities of the device and provide users with a richer and more convenient intelligent experience.

[0038] The object tracking method of this embodiment uses an acoustic sensor to collect regional sound signals and extract the user's footsteps signal from them, which enables the display device to be unrestricted by light conditions and accurately identify and track users even in dimly lit environments or environments with many visual obstacles. Secondly, by analyzing the footsteps signal to determine the user's relative position and walking speed, the display device can adjust the moving direction and speed of the mobile device in real time, keep in sync with the user, and automatically follow the user's movement, enhancing the interactivity and user experience of mobile smart devices anytime and anywhere. In addition, this method also reduces interference with the user's line of sight, and the user does not need to face the camera or other visual tracking devices, thereby improving privacy protection.

[0039] This embodiment has a wide range of application scenarios. It can not only be used for home entertainment, but can also be expanded to the fields of smart services and security monitoring in public places. It provides a new interactive mode for mobile smart devices and greatly enriches the user experience and application scope.

[0040] Combine the following Figure 2 The object tracking method of this embodiment is specifically introduced, which can be executed by the display device 101 and specifically includes the following steps: Step 201: When the user object moves in the target area, a regional sound signal collected by the acoustic sensor is received.

[0041] Specifically, when the user moves in the target area, the acoustic sensor (which can be a microphone array composed of multiple precisely arranged microphones) will synchronously collect sound signals from the area, including environmental noise, conversations, and the user's footsteps, etc. These microphones can capture the subtle differences in sound, including the intensity, frequency, and time difference of the sound reaching each microphone.

[0042] The acoustic sensor then sends the regional sound signal to the display device, and the display device performs further processing after receiving the regional sound signal.

[0043] Step 202: extract the target foot sound signal of the user object from the regional sound signal.

[0044] Specifically, the processor in the display device uses signal processing technology, such as digital signal processing (DSP) and machine learning algorithms, to analyze and process the collected regional sound signal data. The display device analyzes the frequency characteristics of the sound in the regional sound signal, because footsteps usually have a specific frequency range and rhythm pattern, thereby distinguishing the footsteps of the user object from the background noise.

[0045] In this embodiment, in order to accurately distinguish footsteps from background noise, the display device first removes signals beyond the frequency range of footsteps through a bandpass filter. Usually, footsteps are concentrated in the medium and low frequency bands, while environmental noise may contain a wider range of frequency components. Then, using spectrum analysis techniques, such as fast Fourier transform (FFT), the time series sound signal is converted into the frequency domain to reveal the intensity of different frequency components. The display device will identify the frequency peaks and rhythmic patterns unique to footsteps, which are usually manifested as a series of regular impact and friction sounds, in contrast to random environmental noise. In addition, more advanced signal processing techniques, such as wavelet transform, can also be used to analyze sound signals at different time scales to further separate footsteps.

[0046] Finally, the characteristics of footsteps can be used for pattern recognition and classification through machine learning algorithms, such as support vector machines (SVM) or neural networks. These algorithms are able to learn the complex patterns of footsteps based on a large amount of training data and realize automated gait recognition in practical applications.

[0047] Through these technologies, the display device can accurately extract the user's footsteps signal from the complex acoustic environment, providing accurate data support for subsequent user positioning and tracking. This process not only requires efficient algorithms to process large amounts of acoustic data, but also requires real-time performance to ensure that the display device can quickly respond to changes in the user's position, thereby achieving accurate tracking.

[0048] Step 203: Determine the relative position information and walking speed of the user object according to the target footsteps signal.

[0049] The relative position information includes the distance between the user object and the mobile smart device, the position of the user object, etc. Specifically, the distance, direction and moving speed of the user object can be determined according to the footsteps signal of the user object, and the specific implementation method is as follows: 1) To determine the direction, you can use: Microphone array: An array of multiple microphones can capture footsteps from different directions. By analyzing the time difference (TDOA) and phase difference of the sound arriving at each microphone, the direction of the sound source can be determined.

[0050] Beamforming technology: By adjusting the gain of the microphone array, a beam pointing in a specific direction can be formed, thereby enhancing the sound signal from that direction and helping to determine the direction of the sound source.

[0051] 2) To determine the distance, you can use: Sound wave arrival time: By measuring the time it takes for a sound wave to reach the microphone from the sound source, the distance of the sound source can be estimated. This requires knowing the speed of sound in the medium (for example, about 343 m / s in air).

[0052] Sound intensity attenuation: Sound will attenuate during propagation. By measuring the ratio of the received sound intensity to the reference sound intensity, the distance of the sound source can be estimated.

[0053] Echolocation: Similar to the echolocation of bats and dolphins, by emitting sound waves and receiving their echoes, the time it takes for the sound waves to travel back and forth can be measured, thereby calculating the distance.

[0054] 3) To determine the moving speed, you can use: Doppler effect: When the sound source or receiver moves, the frequency of the received sound waves will change. This phenomenon is called the Doppler effect. By measuring the change in frequency, the speed of the sound source can be calculated.

[0055] Time-frequency analysis: By performing short-time Fourier transform (STFT) or wavelet transform on the footstep signal, the time-frequency characteristics of the signal can be analyzed to estimate the movement speed.

[0056] Gait analysis: By analyzing continuous footsteps signals, step length and speed can be estimated, and then the movement speed can be calculated.

[0057] The above specific implementation steps may include: Signal acquisition: Use microphone array to collect footsteps signals. Preprocessing: Perform preprocessing such as noise reduction and gain adjustment on the collected signals. Feature extraction: Extract the characteristics of footsteps, such as frequency, rhythm, intensity, etc. Sound source localization: Use microphone array and beamforming technology to determine the direction of the sound source. Distance estimation: Estimate the distance through sound wave arrival time, sound intensity attenuation or echolocation technology. Speed ​​estimation: Use Doppler effect and gait analysis technology to estimate movement speed. Data fusion: Combine the above information and use statistical and machine learning methods to optimize and calibrate the results.

[0058] Step 204: According to the relative position information and the walking speed, the moving direction and the moving speed of the mobile device are controlled to match those of the user object, so as to track the display device.

[0059] Specifically, after determining the relative position information and walking speed of the user object, the display device will use these data to accurately control the moving direction and speed of the mobile device to achieve real-time tracking of the user object.

[0060] First, the display device will process the footsteps signal collected by the acoustic sensor and obtain the position and speed information as input. Through the built-in control unit, this information is converted into specific movement instructions. The control unit may use a proportional-integral-derivative (PID) controller or more advanced control algorithms, such as fuzzy logic control or adaptive control strategies, to dynamically adjust the movement of the mobile device. These algorithms can calculate the target position and speed that the mobile device needs to achieve based on the real-time position and speed changes of the user object to keep the display device facing the user at all times. The drive motors and wheels on the mobile device will be precisely adjusted according to these instructions to achieve smooth and accurate movement.

[0061] At the same time, the mobile device may also be equipped with additional sensors, such as gyroscopes and accelerometers, which provide feedback on the mobile device's own motion state, helping the control unit to make real-time adjustments and optimizations to ensure that the direction and speed of movement of the mobile device can accurately match the movement of the user object, achieving stable and continuous tracking. Through this closed-loop control mechanism, the display device can flexibly adapt to the movement changes of the user object, and the display device can maintain the best tracking effect regardless of whether the user is walking in a straight line or turning.

[0062] In a specific application, after determining the relative position information of the user object, the display device will compare and analyze this position information with its own current position. If the user is on the left side of the display device, the display device will send a command to the mobile device to move to the left. Similarly, if the user is on the right, in front or behind, the display device will adjust the moving direction of the mobile device accordingly to ensure that it always moves in the direction of the user.

[0063] Secondly, for walking speed matching, the display device will continuously monitor the user's walking speed changes. When the user speeds up, the display device will immediately sense it and increase the speed of the mobile device to ensure that it can keep up with the user. Conversely, when the user slows down, the display device will also reduce the speed of the mobile device to avoid getting too close to the user or falling too far behind.

[0064] In addition, during the tracking process, the display device will continuously make real-time adjustments based on the newly acquired relative position information and walking speed. By continuously adjusting the moving direction and moving speed, the display device can always maintain a suitable distance and relative position with the user, achieving a stable and efficient tracking effect, and providing users with continuous service and convenient use experience.

[0065] Combine the following Figure 3 To further illustrate the object tracking method implemented in this embodiment, the following steps are included: Step 301: In response to a follow-up instruction of the user object, a pre-movement prompt is issued to the user object.

[0066] The following instruction is that the user object requires the mobile smart device to follow him / her. Before the user object is ready to move, the display device will send a pre-movement prompt to the user object through its built-in speaker or display interface.

[0067] The purpose of this step is to let the user know that the device is about to start tracking their movements. This pre-walk prompt may be an audio signal or a visual prompt to ensure that the device is ready to track the user before they start walking. This step is the preliminary stage of obtaining the characteristics of the user's footsteps, laying the foundation for subsequent identity confirmation and tracking.

[0068] In some embodiments, when the user object moves in the target area, it is also necessary to determine whether there are other objects in the surrounding area of ​​the user object; if there are other objects, the user object is prompted to maintain a set distance from the other objects.

[0069] Specifically, when the user object moves in the target area, the display device will monitor the surrounding area of ​​the user object through its acoustic sensor and possibly integrated additional sensors such as cameras to determine whether there are other objects. This step involves real-time analysis of the collected sound signals and video images, using sound recognition technology and image recognition algorithms to distinguish the presence of the user object from other objects.

[0070] If the system determines that there are other objects around the user object, the display device will inform the user object through voice or visual prompts, suggesting that he keep a preset safe distance from these other objects, so as to create a relatively pure environment for the subsequent determination of the footstep characteristics of the user object.

[0071] Such prompts help reduce the interference of other objects' footsteps on the user object's footstep signal recognition, ensuring that the display device can accurately capture and recognize the user object's footstep characteristics, thereby maintaining the accuracy of tracking and the reliability of device response. During this process, the display device may also use sound source separation technology to distinguish the footsteps of different objects to further optimize the tracking effect.

[0072] Step 302 : when the user object moves in the target area according to the pre-movement prompt, a pre-movement sound signal is obtained.

[0073] When the user object starts to walk in the target area according to the pre-walking prompt, the display device uses its acoustic sensor, that is, the microphone array, to obtain the walking sound signals generated by the user's walking. These walking sound signals include the user's footsteps and other possible environmental sounds.

[0074] The display device will monitor and record these sounds in real time, providing the necessary data for subsequent footstep signal recognition and user tracking. This step is a key link in data collection during the tracking process, ensuring that the device can capture the sound characteristics generated when the user moves.

[0075] Step 303: Perform footstep signal recognition on the pre-walking sound signal to obtain footstep features of the user object.

[0076] Among them, footstep features refer to the unique properties and patterns of footstep sound signals generated by human individuals during walking. These features include the rhythm, intensity, frequency distribution, pitch changes, and timing patterns of footsteps, which can reflect information such as the individual's walking style, gait, and step size. Since everyone's walking habits and biomechanical characteristics are different, everyone's footstep features have a certain degree of uniqueness, which makes footstep features can be used as a biometric technology for individual identification. By analyzing and comparing footstep features, the footsteps of different people can be distinguished, and even the footsteps of a specific individual can be identified in an environment where multiple people are walking at the same time. This recognition technology has broad application potential in security monitoring, personal identity verification, and human-computer interaction.

[0077] In this embodiment, the display device uses signal processing technology to analyze and identify these sound signals to extract the characteristics of the user's footsteps. This step involves spectrum analysis of the sound signal and possible pattern recognition algorithms, the purpose of which is to identify the user's unique footstep pattern from the collected sound.

[0078] Through this step, the display device can accurately identify the characteristics of the user's footsteps, which will then be used to identify the user's footsteps from the regional sound signals in the subsequent process, thereby confirming the user's identity and accurately tracking the user. This step is a key technical link in realizing user identity recognition and tracking, which enables the display device to distinguish different users based on footsteps characteristics, thereby providing personalized services and responses.

[0079] Step 304: When the user object moves in the target area, a regional sound signal collected by the acoustic sensor is received.

[0080] This step may refer to step 201 of the above embodiment and will not be described in detail here.

[0081] Step 305: when the footstep sound feature of the user object is confirmed, a target foot sound signal of the user object is extracted from the regional sound signal according to the footstep sound feature.

[0082] Specifically, after confirming the footstep sound characteristics of the user object, the display device uses acoustic signal processing technology to extract the target foot sound signal of the user object from the regional sound signal.

[0083] First, the display device uses its acoustic sensor, a microphone array, to capture all sound signals in the target area, including the user's footsteps and other background noise.

[0084] Then, the display device uses digital signal processing technology, such as bandpass filtering, to remove signals that do not match the footstep characteristics, such as signals that do not match the frequency range of the user object's footsteps, and focuses on those frequency components that match the known footstep characteristics.

[0085] Next, the time domain signal is converted into a frequency domain signal using frequency domain analysis methods such as Fast Fourier Transform (FFT) to further analyze and identify the specific frequency patterns of footsteps. In addition, the display device also uses machine learning algorithms that can automatically identify and distinguish target footstep signals from other background noise or non-target sounds based on previously learned footstep feature patterns. These algorithms lock in signals that match the characteristics of the user's footsteps by analyzing parameters such as the timing pattern, rhythm, and intensity of the sound signal.

[0086] Finally, through sound source localization technology, such as time difference localization (TDOA) or beamforming, the display device can further determine the spatial position of the footsteps signal, ensuring the accuracy and reliability of the target footsteps sound signal extracted from the regional sound signal.

[0087] Through the application of this series of comprehensive technologies, the display device can accurately extract the footsteps signal of a specific user from the regional sound signal in an environment with mixed sounds, providing a solid foundation for subsequent user tracking and interaction.

[0088] In some embodiments, after extracting the target footstep sound signal of the user object, the movement path of the user object can also be determined based on the target footstep sound signal; then, based on the movement path and the planned navigation path, it is judged whether the movement direction of the user object is correct; if the movement direction of the user object is wrong, a prompt indicating the wrong movement direction is issued to the user object.

[0089] Specifically, the display device can first determine the movement path of the user object by analyzing the target footsteps signal collected from the acoustic sensor. This step involves analyzing the time series and frequency characteristics of the footsteps signal to identify the user's footstep rhythm, step length, and direction changes. Through sound source localization technology, such as time difference localization (TDOA) or beamforming, the display device can track the source of the footsteps and infer the user's movement trajectory. This sound-based tracking method makes it possible to accurately capture the user's movement path even when visual information is limited.

[0090] After determining the action path of the user object, the display device will compare the path with the pre-planned navigation path to determine whether the user's action direction is correct. This step requires the display device to have a path planning and navigation algorithm that can generate a navigation path based on the user's target location or predetermined route. By comparing the user's actual action path with the planned navigation path, the display device can evaluate whether the user's movement deviates from the predetermined direction, thereby providing the user with accurate navigation information.

[0091] If the display device determines that the user's direction of movement is wrong, that is, the user's path of movement is inconsistent with the planned navigation path, it will issue a prompt to the user object that the direction of movement is wrong. This step involves user interface design. The display device will issue instructions to the user to correct the direction of movement through voice prompts, visual signals or vibration feedback. This real-time feedback mechanism can help users adjust their movement direction in time to ensure that they can move along the correct path and avoid getting lost or deviating from the target. This intelligent prompt system improves the user experience and allows users to easily navigate in complex environments or unfamiliar areas.

[0092] If the display device determines that the user's movement direction is correct, it continues to track the user object.

[0093] Step 306: Determine the relative position information and walking speed of the user object according to the target footsteps signal.

[0094] This step may refer to step 203 and will not be described in detail here.

[0095] Step 307: According to the relative position information and the walking speed, the moving direction and the moving speed of the mobile device are controlled to match those of the user object, so as to track the display device.

[0096] This step may refer to step 204 and will not be described in detail here.

[0097] Step 308: confirming the current location information in response to the user object's instruction to cancel following.

[0098] Specifically, when the user object issues a command to release the following, the display device will immediately respond to this command and first confirm the current location information of the mobile device. This step involves positioning technology, such as GPS positioning, indoor positioning system or sensor-based positioning algorithm, to determine the precise location of the mobile device in space. Confirming the location information is the basis for subsequent steps, ensuring that the mobile device can accurately know where it is and provide necessary data for determining the return route.

[0099] Step 309: Determine a return route based on the current location information and the location information of the charging station.

[0100] After confirming the current location information of the mobile device, the display device will determine a return route based on this information and the known location information of the charging station. This step requires a path planning algorithm, which can calculate the shortest or optimal path from the current location to the charging station, taking into account possible obstacles, terrain restrictions and other navigation factors. The path planning algorithm generates a detailed route to guide the mobile device how to return to the charging station safely and efficiently.

[0101] Step 310: Control the mobile device to move toward the charging station according to the return route.

[0102] After determining the homing route, the display device will control the mobile device to move toward the charging station according to this route. This step involves a mobile control algorithm, which can convert the route generated by the path planning algorithm into specific movement instructions, such as steering angle, speed, and acceleration. The drive system on the mobile device will execute these instructions, adjust the speed and direction of the wheels, and move the mobile device along the planned route to the charging station. During this process, the mobile device may also use an obstacle avoidance algorithm to deal with unexpected obstacles that may appear along the way to ensure the safe completion of the homing mission.

[0103] In some embodiments, the implementation process of the method also includes: first, obtaining a character video image of the user object when walking through the camera; then predicting the estimated movement direction of the user object based on the character video image; and finally controlling the movement direction of the mobile device based on the estimated movement direction.

[0104] Specifically, the display device first uses its built-in high-definition camera to obtain a video image of the user object walking in the target area. During this process, the camera will capture the user's actions in real time, record their movement trajectory and posture changes. These video image data not only contain the user's location information, but may also contain dynamic information such as the user's movements and speed, providing rich visual data for the display device. These data will then be transmitted to the processor of the display device for subsequent analysis and processing to achieve accurate tracking of the user's actions.

[0105] After acquiring a person video image of the user object, the display device uses computer vision technology and machine learning algorithms to predict the user's estimated direction of movement based on these images. This step involves analyzing the user's posture, steps, and movement trends in the video images, as well as learning possible historical movement patterns. Through these analyses, the display device is able to predict the direction the user is about to move, even if the user has not actually moved to the new location. This predictive ability allows the display device to prepare in advance and adjust its tracking strategy to ensure that the user is always in the center of the camera's field of view, providing a smoother and more natural tracking experience.

[0106] Based on the estimated direction of movement predicted from the video image of the person, the display device controls the direction of movement of the mobile device to maintain its relative position with the user object. This step involves converting the predicted direction of movement into specific movement instructions, which are sent to the drive system of the mobile device, such as motors and wheels. The mobile device adjusts its movement direction based on these instructions to ensure that the display device always faces the user object, regardless of whether the user is walking in a straight line or turning. This real-time adjustment and responsiveness allows the display device to flexibly adapt to the user's movement changes, achieve accurate and continuous tracking, and provide users with an interactive experience that always responds in real time.

[0107] The embodiment of the present invention provides a new object tracking method, which is specially designed for mobile smart devices such as girlfriend phones or smart screens.

[0108] First, the display device of the mobile smart device will issue a pre-movement prompt after the user object issues a follow-up instruction, and collect pre-movement sound signals when the user starts walking, and identify the user's footstep characteristics through these signals. After confirming the user's footstep characteristics, the display device can accurately extract the user's target foot sound signal from the complex regional sound signals. Using these signals, the display device can determine the user's relative position information and walking speed, and control the moving direction and speed of the mobile device accordingly to maintain synchronization with the user object and realize the tracking function.

[0109] In addition, the display device also obtains video images of the user walking through the camera, and predicts the user's estimated direction of movement based on these images, thereby controlling the direction of movement of the mobile device. The display device also determines the user's movement path based on the user's footsteps signal, and compares it with the preset navigation path to determine whether the user's movement direction is correct. If the user deviates from the predetermined route, the display device will issue a prompt to the user that the movement direction is wrong, and guide the user back to the correct path.

[0110] Finally, when the user issues a command to cancel the follow-up, the display device will confirm the current location information, determine the return route based on the current location and the location information of the charging station, and control the mobile device to return to the charging station safely. The entire process involves multiple technical links such as sound signal processing, image recognition, path planning and navigation prompts, and embodies an intelligent and automated tracking and navigation solution that aims to improve user experience and ensure that mobile smart devices can flexibly and accurately respond to users' mobility and navigation needs.

[0111] The method provided in the above embodiment can be executed by a display device, which is an electronic device. The electronic device in the embodiment of the present invention is described below from the perspective of hardware processing. Figure 4 , is a schematic diagram of a physical device structure of an electronic device in an embodiment of the present invention.

[0112] It should be noted that Figure 4 The structure of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0113] like Figure 4 As shown, the electronic device includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 402 or the program loaded from the storage part 408 to the random access memory (RAM) 403, such as executing the method described in the above embodiment. In RAM 403, various programs and data required for system operation are also stored. CPU 401, ROM 402 and RAM 403 are connected to each other through bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0114] The following components are connected to the I / O interface 405: an input section 406 including an audio input device, a button switch, etc.; an output section 407 including a liquid crystal display (LCD) and an audio output device, an indicator light, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed so that a computer program read therefrom is installed into the storage section 408 as needed.

[0115] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 409, and / or installed from a removable medium 411. When the computer program is executed by the central processing unit (CPU) 401, various functions defined in the present invention are performed.

[0116] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0117] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings.

[0118] Specifically, the electronic device of this embodiment includes a processor and a memory, the memory is coupled to one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and one or more processors call the computer instructions to enable the electronic device to execute the method provided by the above embodiment.

[0119] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above storage medium carries one or more computer programs, and when the above one or more computer programs are executed by a processor of the electronic device, the electronic device implements the method provided in the above embodiment.

[0120] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

[0121] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0122] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. An object tracking method, characterized in that: A display device applied to a mobile smart device, the mobile smart device further comprising a mobile device and an acoustic sensor, the display device being arranged on the mobile device, the acoustic sensor being connected to the display device, the method comprising: When the user object moves in the target area, receiving the regional sound signal collected by the acoustic sensor; extracting a target foot sound signal of the user object from the regional sound signal; Determine the relative position information and walking speed of the user object according to the target footsteps signal; According to the relative position information and the walking speed, the moving direction and the moving speed of the mobile device are controlled to keep matching with the user object, so as to track the display device.

2. The method according to claim 1, characterized in that The step of extracting the target foot sound signal of the user object from the regional sound signal comprises: When the footstep sound feature of the user object is confirmed, the target foot sound signal is extracted from the regional sound signal according to the footstep sound feature.

3. The method according to claim 2, characterized in that Before the step of receiving the regional sound signal collected by the acoustic sensor when the user object walks in the area, the method further includes: In response to the following instruction of the user object, issuing a pre-movement prompt to the user object; When the user object moves in the target area according to the pre-movement prompt, obtaining a pre-movement sound signal; Perform footstep signal recognition on the pre-walking sound signal to obtain footstep features of the user object.

4. The method according to claim 3, characterized in that Before the step of obtaining the pre-movement sound signal, the method further includes: Determine whether there are other objects within the surrounding area of ​​the user object; If so, the user object is prompted to maintain a set distance from the other object.

5. The method according to any one of claims 1 to 4, characterized in that: The mobile smart device also includes a camera, and the method further includes: Acquire a video image of the user object when it is moving through the camera; Predicting the estimated movement direction of the user object according to the character video image; The moving direction of the mobile device is controlled according to the estimated moving direction.

6. The method according to claim 1, characterized in that After the step of extracting the target foot sound signal of the user object from the regional sound signal, the method further includes: Determining a movement path of the user object according to the target footsteps signal; According to the action path and the planned navigation path, determining whether the action direction of the user object is correct; If not, a prompt indicating that the action direction is wrong is sent to the user object.

7. The method according to claim 1, characterized in that After the step of controlling the moving direction and the moving speed of the mobile device to match the user object according to the relative position information and the walking speed so as to track the display device, the method further includes: In response to a follow-up release instruction from the user object, confirming current location information; Determine a return route according to the current location information and the location information of the charging station; The mobile device is controlled to move toward the charging station according to the return route.

8. A mobile smart device, characterized in that: The invention comprises a display device, a mobile device and an acoustic sensor, wherein the display device is arranged on the mobile device, and the acoustic sensor is connected to the display device; the display device comprises one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the display device to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed on a display device of a mobile smart device, the display device is caused to execute the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that When the computer program product is run on a display device of a mobile smart device, the display device is enabled to execute the method according to any one of claims 1 to 7.

Images