Mobile terminal multi-sensory linkage feedback method and system based on real scene

Abstract

The application discloses a mobile terminal multi-sensory linkage feedback method and system based on a real scene. A multi-modal sensor built in a mobile terminal is used to collect multi-dimensional scene characteristic parameters for representing a real environment state, and the scene characteristic parameters are preprocessed to generate standardized scene characteristic data. Based on the scene characteristic data, a scene recognition model is used to determine a scene type currently located, and further to recognize corresponding scene detail parameters. According to the scene type and the scene detail parameters, multi-sensory feedback parameters of vision, hearing and touch are generated from a preset multi-sensory linkage feedback strategy, and a corresponding sensory feedback execution unit is synchronously driven to execute linkage feedback. When it is detected that the scene characteristic parameters change, the multi-sensory feedback parameters are updated in real time, so that dynamic adaptation of multi-sensory feedback to real scene changes is realized. The immersion, adaptability and practicability of the mobile terminal in different use scenes are improved.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction control technology for smart terminals, and in particular to a multi-sensory linkage feedback method and system for mobile terminals based on real-world scenarios. Background Technology

[0002] With the continuous improvement of mobile terminal hardware performance, mobile terminals such as smartphones are no longer limited to traditional information display and communication functions. Their usage scenarios are gradually extending to more complex and varied real-world environments. In daily use, users are often in different real-world scenarios, such as walking outdoors, commuting, public places like shopping malls, or low-light environments at night. These real-world scenarios have significant differences in lighting conditions, environmental noise, and user movement, which places higher demands on the interaction methods and feedback forms of mobile terminals.

[0003] In existing technologies, the impact of changes in the real-world environment on the user experience of mobile terminals is typically addressed using a single-dimensional perception and feedback approach. For example, a light sensor can be used to automatically adjust screen brightness to adapt to different lighting conditions; a microphone can be used to collect ambient noise and perform noise reduction processing to improve voice call quality; or a linear motor can output preset vibration feedback when a notification is received. These technical solutions mostly optimize only a single sensory dimension independently, with each sensory feedback being isolated and lacking synergy, making it difficult to create a holistic perceptual experience that matches the real-world scenario.

[0004] To improve user experience, some existing solutions attempt to introduce combinations of multiple sensory feedback in specific applications or fixed modes. For example, in game mode, simultaneously increasing the screen refresh rate, enhancing sound effects, and configuring specific vibration effects. However, such multi-sensory combinations usually rely on preset triggering conditions, such as the launch of a specific application or the user manually activating the mode. The feedback parameters are fixed values ​​set in advance, which cannot be dynamically adjusted according to real-time changes in the real scene, nor can they distinguish different detailed states under the same scene type, making the adaptation method rather rigid.

[0005] Furthermore, existing multi-sensory feedback solutions generally suffer from limited recognition dimensions at the scene recognition level. Most solutions rely on only a single or limited number of environmental parameters for judgment, making it difficult to accurately reflect the comprehensive characteristics of real-world scenes. In particular, they cannot effectively distinguish details such as changes in environmental intensity and user behavior within the same scene, resulting in feedback results that do not match the actual usage environment and may even interfere with the user.

[0006] Furthermore, existing technologies often lack a unified linkage control mechanism in the execution of multi-sensory feedback. Visual, auditory, and tactile feedback are often triggered separately by different modules, making it difficult to maintain consistency in activation timing and feedback intensity. This can easily lead to asynchronous feedback, thereby weakening the overall continuity and immersion of the experience. At the same time, existing solutions generally ignore individual user differences and fail to effectively incorporate user preferences for different sensory feedback, resulting in low adaptability.

[0007] In summary, existing technologies lack a solution that can identify multi-dimensional scene features in a comprehensive manner for real-world scenarios, and generate and dynamically adjust multi-sensory interactive feedback based on scene type and details. This makes it difficult to simultaneously ensure immersive experience, adaptability, and actual usage effect in complex and ever-changing real-world usage environments.

[0008] Therefore, existing technologies still need to be improved. Summary of the Invention

[0009] In view of the shortcomings of the existing technologies, there is an urgent need to provide a new multi-sensory feedback technology solution for mobile terminals. This solution should be able to comprehensively perceive and identify environmental conditions in real-world usage scenarios, and on this basis, achieve unified and coordinated control of multiple sensory feedbacks. This would avoid problems such as fragmented sensory feedback, rigid scene adaptation, and asynchronous feedback in existing technologies. Furthermore, this solution should be able to dynamically adjust feedback parameters according to changes in the real-world scenario and take into account the usage preferences of different users, thereby improving the adaptability of mobile terminals in complex and ever-changing real-world scenarios and enhancing the overall user experience.

[0010] The technical solution of the present invention is as follows: This invention provides a multi-sensory interactive feedback method for mobile terminals based on real-world scenarios, comprising the following steps: The mobile terminal uses a built-in multimodal sensor to collect multidimensional scene feature parameters to characterize the state of the real environment, and preprocesses the scene feature parameters to generate standardized scene feature data. Based on the standardized scene feature data, the scene recognition model is used to determine the current scene type, and at least one scene detail parameter corresponding to the scene type is further identified. Based on the scene type and the scene detail parameters, multi-sensory feedback parameters are generated from a preset multi-sensory linkage feedback strategy. The multi-sensory feedback parameters include at least visual feedback parameters, auditory feedback parameters, and tactile feedback parameters. The multi-sensory feedback parameters are synchronously sent to the corresponding sensory feedback execution unit to drive the mobile terminal to execute multi-sensory linkage feedback that matches the current real scene; When it is detected that the scene feature parameters have changed, the multi-sensory feedback parameters are regenerated based on the changed scene feature parameters, and the multi-sensory linkage feedback is synchronously updated.

[0011] In one embodiment, the multi-modal sensors at least include a light sensor, a sound sensor, a motion state sensor, and a positioning sensor, and the scene feature parameters at least include an illumination intensity parameter, an ambient noise parameter, a motion state parameter, and a position information parameter.

[0012] In one embodiment, preprocessing the scene feature parameters includes at least one of the following operations: filtering, normalization, or abnormal data rejection.

[0013] In one embodiment, the scene recognition model is a scene feature recognition model trained based on a machine learning algorithm or a deep learning algorithm, and is used to output the corresponding scene type and the scene detail parameters.

[0014] In one embodiment, the scene detail parameters are used to characterize the environmental intensity level, the environmental change state, or the user behavior state under the same scene type, and are used to refine and adjust the multi-sensory feedback parameters.

[0015] In one embodiment, the multi-sensory feedback parameters are generated by a unified multi-sensory linkage control strategy, so that the visual feedback, the auditory feedback, and the tactile feedback are coordinated and consistent in the feedback timing and the feedback intensity.

[0016] In one embodiment, when the change amount of the scene feature parameters exceeds a preset threshold, it triggers the regeneration and synchronous update of the multi-sensory feedback parameters.

[0017] In one embodiment, when generating the multi-sensory feedback parameters, the sensory preference parameters预先 configured by the user are also combined to perform personalized adjustment on the multi-sensory feedback parameters.

[0018] On the other hand, the present invention also provides a mobile terminal multi-sensory linkage feedback system based on a real scene, including: A scene feature acquisition module, configured to acquire multi-dimensional scene feature parameters; A scene recognition and analysis module, configured to determine the scene type and the scene detail parameters; A multi-sensory linkage control module, configured to generate multi-sensory feedback parameters; A sensory feedback execution module, configured to synchronously execute the multi-sensory linkage feedback; Wherein, each module is cooperatively configured to execute the method described in any one of the above.

[0019] In another aspect, the present invention further provides a mobile terminal, including a processor and a memory. A computer program is stored in the memory. When the computer program is executed by the processor, the mobile terminal executes the method described in any one of the above.

[0020] In summary, through the comprehensive collection and analysis of multi-dimensional environmental features in the real scenario, the present invention realizes the accurate recognition of the current usage scenario of the mobile terminal, and on this basis, constructs a multi-sensory linkage feedback mechanism with the scenario type and scenario details as the core. Compared with the existing feedback methods mainly based on single-sense or fixed trigger conditions, the present invention incorporates visual, auditory, and tactile feedback into a unified linkage control framework, enabling multiple sensory feedbacks to be coordinated in time and intensity, thus forming an overall feedback effect that matches the real scenario.

[0021] Meanwhile, the present invention introduces a dynamic adjustment mechanism based on the changes in scenario features. When the real environment changes, it can timely update the multi-sensory feedback parameters, making the feedback effect continuously fit the actual usage scenario and avoiding the problem of experience mismatch caused by environmental changes. In addition, by combining the sensory preference parameters pre-configured by the user, the present invention further improves the adaptability to different user usage habits and perception differences while ensuring scenario adaptability.

[0022] Therefore, the technical solution proposed by the present invention realizes the multi-sensory collaborative feedback of the mobile terminal in complex and changing real scenarios without increasing additional hardware costs, taking into account the immersive experience, adaptability, and actual usage effects, and has good engineering feasibility and application promotion value.

[0023] Compared with the prior art, the present invention does not simply superimpose or juxtapose multiple sensory feedback methods. Instead, with the real scenario as the core, through the comprehensive recognition of multi-dimensional scenario features, it constructs a multi-sensory linkage feedback mechanism driven by the scenario type and scenario details, changing the response logic of the mobile terminal to environmental changes from the overall interaction mode, thus bringing multiple beneficial effects.

[0024] First of all, by distinguishing the scenario type and scenario details, the present invention enables the multi-sensory feedback to be finely adjusted according to the changes in the real environment, rather than only making a coarse-grained response when the scenario switches. The prior art usually can only trigger feedback based on a single environmental parameter or fixed conditions. The present invention can dynamically adjust the visual, auditory, and tactile feedback parameters according to the changes in environmental intensity or the different user behavior states under the same scenario type, so as to obtain a more natural and fitting feedback experience in actual use. This feedback adjustment method based on scenario details enables the user to still perceive the continuous response of the terminal to environmental changes without any active operation, which is an experience improvement difficult to anticipate in the prior art.

[0025] Secondly, this invention achieves coordinated consistency in both time and intensity of visual, auditory, and tactile feedback through a unified multi-sensory linkage control strategy, avoiding the problem of separate and asynchronous triggering of sensory feedback in existing technologies. In practice, it has been found that when multiple sensory feedbacks can be presented collaboratively within a similar timeframe, the user's efficiency in understanding the feedback information and the stability of their perception are significantly improved. This synergistic effect cannot be achieved simply by increasing the types of feedback, but rather relies on a unified generation and synchronous execution control mechanism. Through this linkage mechanism, this invention achieves a qualitative leap in the overall interactive experience without adding extra hardware, demonstrating remarkable unexpectedness.

[0026] Furthermore, this invention introduces a dynamic update mechanism based on changes in scene features, enabling multi-sensory feedback to continuously adapt to changes in the real-world environment, rather than remaining unchanged after the initial trigger. In existing technologies, even if multi-sensory feedback schemes exist, their feedback parameters are mostly preset fixed values. Once the environment changes, the feedback effect often lags or becomes mismatched. This invention, through real-time monitoring and judgment of changes in scene features, promptly regenerates and synchronously updates multi-sensory feedback parameters, ensuring that the feedback always maintains a correspondence with the real-world scene, thereby significantly improving the adaptability of mobile terminals in complex and changing environments.

[0027] Furthermore, this invention integrates personalized user preference parameters into the multi-sensory interactive feedback, enabling the feedback effect in the same real-world scenario to be adjusted according to the perceptual differences of different users. Compared with the existing technology's "uniform parameters, uniform experience" approach, this invention ensures accurate scenario adaptation while taking into account individual user differences, avoiding the problem of multi-sensory feedback interfering with some users. This design, which introduces personalized adjustment into system-level scenario adaptation, makes multi-sensory feedback both consistent with scenario logic and possesses good user adaptability; its overall effect is beyond the expectations of existing technologies.

[0028] Finally, the technical solution described in this invention is entirely based on existing mobile terminal sensor and feedback hardware, achieving multi-sensory collaboration and dynamic adaptation at the system level without adding any additional hardware modules. This not only significantly improves the interactive experience but also possesses good engineering feasibility and promotional value, enabling a low-cost overall upgrade of existing technical solutions in actual products. Attached Figure Description

[0029] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 A flowchart illustrating the method steps of a mobile terminal multi-sensory linkage feedback method based on real-world scenarios provided by the present invention; Figure 2 The flowchart of a multi-sensory linkage feedback method for a mobile terminal based on a real-world scenario provided by the present invention; Figure 3 The system structure block diagram of a multi-sensory linkage feedback system for a mobile terminal based on a real-world scenario provided by the present invention. Detailed implementation manners

[0030] To make the objectives, technical solutions and effects of the present invention clearer and more definite, the present invention will be further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. The embodiments of the present invention will be introduced below with reference to the accompanying drawings.

[0031] A multi-sensory linkage feedback method for a mobile terminal based on a real-world scenario provided by an embodiment of the present invention is described by taking a smart phone as an example, but the protection scope of the present invention is not limited thereto.

[0032] In this embodiment, the mobile terminal is pre-integrated with multi-modal sensors, and a multi-sensory linkage feedback function is deployed at the system level. The multi-sensory linkage feedback function can run as a system service resident, and the user can choose to enable or disable it through the settings interface. Please refer to Figure 1 、 Figure 2 , which includes the following steps: S1. Collect multi-dimensional scene feature parameters for characterizing the state of the real environment through the multi-modal sensors built in the mobile terminal, and preprocess the scene feature parameters to generate standardized scene feature data; During the normal use of the mobile terminal, multi-dimensional scene feature parameters for characterizing the state of the real environment are collected in real time through the multi-modal sensors built in the terminal. The multi-modal sensors at least include a light sensor, a sound sensor, a motion state sensor, and a positioning sensor, and the corresponding collected scene feature parameters at least include environmental light intensity, environmental noise level, terminal motion state, and location information, etc.

[0033] To improve the stability and accuracy of subsequent scene recognition, preprocessing operations are performed on the collected original scene feature parameters. Specifically, filtering processing can be performed on the light intensity and noise data to eliminate abnormal fluctuations caused by sensor jitter or instantaneous interference; normalization processing is performed on the scene feature parameters of different dimensions to make each parameter within a unified magnitude range; and obvious outliers are removed or corrected when detected, so as to generate standardized scene feature data with a unified structure and directly used for analysis.

[0034] Furthermore, the method further includes: S2. Based on the standardized scene feature data, the scene recognition model is used to determine the current scene type, and at least one scene detail parameter corresponding to the scene type is further identified; The standardized scene feature data obtained above is input into a pre-trained scene recognition model, which then analyzes and judges the current real-world scene. The scene recognition model can be built based on machine learning or deep learning algorithms and is used to comprehensively output the current scene type from multi-dimensional scene features, such as an outdoor walking scene, a commuting scene, an indoor public place scene, or a stationary usage scene.

[0035] Based on the determined scene type, at least one scene detail parameter corresponding to the scene type is further identified. This scene detail parameter characterizes the environmental intensity level, change state, or user behavior state within the same scene type, such as changes in light intensity, environmental noise level, or the change in the terminal's state from stationary to moving. By introducing scene detail parameters, the system can distinguish different usage states within the same scene type, providing a basis for the refined generation of subsequent feedback parameters.

[0036] Furthermore, the method also includes: S3. Based on the scene type and the scene detail parameters, generate multi-sensory feedback parameters from a preset multi-sensory linkage feedback strategy. The multi-sensory feedback parameters include at least visual feedback parameters, auditory feedback parameters, and tactile feedback parameters. After completing the scene type and scene detail identification, corresponding multi-sensory feedback parameters are generated from a preset multi-sensory linkage feedback strategy based on the scene type and scene detail parameters. The multi-sensory linkage feedback strategy is configured with the scene as the core and is used to uniformly describe the combination of visual, auditory, and tactile feedback in a specific scene and its detailed state.

[0037] The generated multi-sensory feedback parameters include at least visual, auditory, and tactile feedback parameters. Visual feedback parameters can be used to control screen brightness, contrast, display effects, or prompting methods; auditory feedback parameters can be used to control volume, sound effect enhancement, or voice prompt strategies; and tactile feedback parameters can be used to control vibration intensity, frequency, or rhythm. These various feedback parameters are not generated independently but are collaboratively determined through a unified multi-sensory linkage control strategy to ensure consistency in the overall experience across different sensory feedbacks.

[0038] Furthermore, the method also includes: S4. The multi-sensory feedback parameters are synchronously sent to the corresponding sensory feedback execution unit to drive the mobile terminal to execute multi-sensory linkage feedback that matches the current real scene. The generated multi-sensory feedback parameters are synchronously sent to the corresponding sensory feedback execution units, and the sensory feedback execution units drive the mobile terminal to execute multi-sensory linkage feedback matching the current real scenario. The sensory feedback execution units can respectively correspond to the screen display unit, the audio output unit, and the vibration feedback unit.

[0039] During the execution process, a synchronization control mechanism is adopted to ensure the consistency of the triggering timings of visual, auditory, and tactile feedbacks, avoiding an obvious time difference caused by the separate responses of each feedback module, so as not to affect the coherence of the overall experience. Through the above synchronous execution method, a unified and coordinated multi-sensory feedback effect is formed in the user's perception.

[0040] Further, the method further includes: S5. When it is detected that the scenario feature parameters change, the multi-sensory feedback parameters are regenerated based on the changed scenario feature parameters, and the multi-sensory linkage feedback is synchronously updated.

[0041] During the execution of the multi-sensory linkage feedback, the change situation of the scenario feature parameters is continuously monitored. When it is detected that the change amount of the scenario feature parameters exceeds a preset threshold, it is determined that the current real scenario has changed, and the scenario recognition and analysis process is re-executed based on the changed scenario feature parameters.

[0042] After re-determining the scenario type or scenario detail parameters, the multi-sensory feedback parameters are regenerated, and the currently executed multi-sensory linkage feedback is synchronously updated, so that the feedback effect can continuously fit the changed real scenario. Through the above dynamic update mechanism, it is avoided that the multi-sensory feedback remains unchanged after the environment changes, thereby improving the adaptability of the mobile terminal in a complex and changeable real environment.

[0043] Through the above implementation manners, the present invention realizes multi-sensory linkage feedback based on the real scenario without increasing additional hardware costs, and can be dynamically adjusted according to the change of the scenario, having good engineering feasibility and practical application value.

[0044] In a further embodiment, the multi-modal sensors built in the mobile terminal at least include a light sensor, a sound sensor, a motion state sensor, and a positioning sensor. Each type of sensor is uniformly connected to the scenario feature acquisition module at the system level, and the scenario feature acquisition module coordinates, manages, and schedules the acquisition of sensing data from different sources.

[0045] The light sensor is used to collect light intensity information of the current real environment. The obtained light intensity parameters can be used to reflect the ambient brightness level and its changing trend. In specific implementation, the light sensor can periodically output light intensity values ​​according to a preset sampling frequency. When a significant change in light intensity is detected in a short period of time, the scene feature acquisition module can appropriately increase the sampling frequency to improve the response sensitivity to changes in light intensity.

[0046] The sound sensor is used to collect environmental noise information in the real environment, specifically including noise intensity parameters. By analyzing and processing the environmental sound signals, noise intensity data reflecting the level of environmental noise can be obtained. In practical applications, the noise intensity parameter can serve as one of the important criteria for determining whether the current usage scenario is a public place, a commuting environment, or a quiet environment.

[0047] The motion state sensor is used to collect motion state parameters of the mobile terminal, which may include information such as changes in the terminal's acceleration, speed, or motion trend. By analyzing the motion state parameters, it can be determined whether the mobile terminal is currently stationary, moving slowly, or moving rapidly, thus providing behavioral-level reference information for scene type and scene details identification.

[0048] The positioning sensor is used to collect location information parameters of the mobile terminal, which may include geographical location information or location change information. In specific implementations, the location information parameters can be used in conjunction with motion state parameters to help determine whether there has been a change in spatial location in the current scene, such as switching from an indoor environment to an outdoor environment, or entering a mobile scene from a fixed location.

[0049] During the scene feature parameter acquisition process, the illumination intensity parameter, environmental noise parameter, motion state parameter, and location information parameter are not acquired in isolation, but rather as multi-dimensional scene features within the same time window. The scene feature acquisition module can perform time alignment processing on the above-mentioned parameters to form a multi-dimensional scene feature parameter set that can comprehensively reflect the current real-world environmental state.

[0050] By employing a multi-dimensional scene feature acquisition method that includes at least the aforementioned light intensity parameters, environmental noise parameters, motion state parameters, and location information parameters, this implementation method can fully characterize the features of a real-world scene from both environmental perception and user behavior perspectives. This provides a reliable data foundation for the accurate identification of subsequent scene types and details, thereby ensuring the accuracy and stability of multi-sensory interactive feedback generation.

[0051] In a further embodiment, preprocessing the scene feature parameters includes at least one of the following operations: filtering, normalization, or outlier removal.

[0052] The illumination intensity parameters, environmental noise parameters, motion state parameters, and location information parameters acquired by the scene feature acquisition module are first input into the scene feature preprocessing module for unified processing. Because different sensors have varying sampling frequencies, data volumes, and data stability, direct analysis can easily lead to fluctuations or misjudgments in scene recognition results. Therefore, it is necessary to preprocess these multi-dimensional scene feature parameters at the system level.

[0053] Specifically, the preprocessing process includes at least data denoising, time synchronization, and scale unification.

[0054] Among them, data denoising is used to filter out instantaneous outliers in illumination intensity parameters and environmental noise parameters to avoid interference from short-term environmental disturbances on the overall scene judgment; time synchronization is used to align parameter data from different sensors according to the same time window to ensure that multi-dimensional scene features are comprehensively analyzed under the same real-world conditions; scale unification processing is used to convert parameter data of different dimensions into comparable feature forms for subsequent fusion calculations.

[0055] After preprocessing, the scene feature preprocessing module inputs various processed scene feature parameters into the scene fusion analysis module. The scene fusion analysis module, based on a preset fusion strategy, performs joint analysis on the multi-dimensional scene feature parameters to generate a fused scene feature result that characterizes the current state of the real environment.

[0056] Through the preprocessing and fusion analysis of multi-dimensional scene feature parameters described above, this implementation method can effectively reduce the impact of single sensor errors on scene judgment results at the system level, improve the stability and consistency of scene feature expression, and provide a reliable foundation for subsequent determination of scene type and scene details.

[0057] In a further embodiment, the scene recognition model is a scene feature recognition model trained based on machine learning algorithms or deep learning algorithms, used to output the corresponding scene type and the scene detail parameters.

[0058] In this embodiment, the fused scene feature results output by the scene fusion analysis module are input to the scene recognition module, which then determines the current real-world scene. The scene recognition module can analyze the fused scene feature results based on preset scene recognition rules or models to determine the scene type corresponding to the current scene and the scene details related to that scene type.

[0059] The scene type describes the overall category of the current real-world environment, such as whether it is an indoor environment, an outdoor environment, a commuting scenario, or a relatively quiet usage scenario; the scene details are used to further depict the specific state differences under the same scene type, such as changes in light intensity, changes in environmental noise level, and whether the user is currently in a moving state.

[0060] In practical implementation, the scene recognition module can analyze the combination relationship between the feature parameters in the fused scene feature results to determine whether the comprehensive features of the current real environment meet the judgment conditions of a certain scene type. After determining the scene type, it can further identify the corresponding scene details based on the change range and trend information of the feature parameters.

[0061] Through the above-described process of identifying scene types and details, this implementation method can achieve more refined and stable identification of real-world scenes without relying on the judgment results of a single sensor. This improves the system's understanding of the real environment from coarse-grained judgment to a hierarchical scene cognition result, providing clear and executable input basis for the subsequent generation of multi-sensory linkage feedback strategies.

[0062] In a further embodiment, the scene detail parameters are used to characterize the environmental intensity level, environmental change state, or user behavior state under the same scene type, and are used to refine and adjust the multi-sensory feedback parameters.

[0063] In this embodiment, after the scene recognition module completes the identification of the scene type and scene details of the current real-world scene, it inputs the corresponding scene recognition results into the linkage feedback generation module. The linkage feedback generation module is used to determine a matching multi-sensory linkage feedback scheme based on the scene type and scene details.

[0064] Specifically, the linkage feedback generation module pre-stores multiple sets of feedback strategy configurations corresponding to different scene types and scene details. Each set of feedback strategy configurations includes at least one or more of visual feedback parameters, auditory feedback parameters, and tactile feedback parameters. The feedback strategy configurations are not simple fixed combinations, but are constructed based on the correlation between scene features, so that the multiple sensory feedbacks form a unified feedback style as a whole.

[0065] When generating a multi-sensory linkage feedback scheme, the linkage feedback generation module first selects a basic feedback strategy that matches the identified scene type from the preset feedback strategy configuration; then, it adjusts the various sensory feedback parameters in the basic feedback strategy in combination with scene detail information, thereby generating a target multi-sensory linkage feedback scheme that is adapted to the current real scene.

[0066] For example, in the same scene type, when the scene details reflect changes in environmental lighting or noise levels, the associated feedback generation module can correspondingly adjust the brightness or color representation of the visual feedback, and simultaneously coordinate and adjust the intensity of the auditory feedback or tactile feedback, so that the finally generated multi-sensory associated feedback scheme can maintain consistency in the overall perception.

[0067] Through the above method, this embodiment changes the generation process of multi-sensory feedback from the independent decision-making of single senses to the overall associated decision-making based on the scene type and scene details, effectively avoiding the problem of fragmentation among multi-sensory feedbacks in the prior art.

[0068] In a further embodiment, the multi-sensory feedback parameters are generated by a unified multi-sensory associated control strategy, so that the visual feedback, auditory feedback, and tactile feedback are coordinated and consistent in the feedback timing and feedback intensity.

[0069] The target multi-sensory associated feedback scheme generated by the associated feedback generation module is transmitted to the multi-sensory feedback control module. The multi-sensory feedback control module is used to uniformly schedule and coordinately control the visual feedback unit, auditory feedback unit, and tactile feedback unit according to the target multi-sensory associated feedback scheme.

[0070] Specifically, before executing the feedback output, the multi-sensory feedback control module first analyzes various feedback parameters in the target multi-sensory associated feedback scheme, and determines the time relationship and intensity relationship between different sensory feedbacks. By uniformly managing the feedback trigger timing at the system level, it is avoided that the sensory feedbacks are misaligned or interfere with each other during the output process.

[0071] During the execution process, the multi-sensory feedback control module can synchronously or time-divisionally trigger the corresponding visual feedback, auditory feedback, and tactile feedback according to the target multi-sensory associated feedback scheme. Among them, synchronous triggering is used in scenarios where the overall perception effect needs to be emphasized, so that multiple sensory feedbacks are output within the same time window; time-division triggering is used in scenarios where the feedback effect needs to be presented hierarchically, so that different sensory feedbacks are output in a predetermined order in sequence, thereby enhancing the hierarchy of the feedback.

[0072] In addition, when it is detected that the real scene changes during the feedback output process, the multi-sensory feedback control module can dynamically adjust or switch the multi-sensory associated feedback scheme being executed according to the updated scene recognition result, so as to ensure that the output multi-sensory feedback continuously remains consistent with the current real scene.

[0073] Through the above collaborative output mechanism of multi-sensory feedback, this embodiment can achieve the unified control and coordinated execution of multiple sensory feedbacks at the system level, enabling users to obtain a coherent, natural, and stable multi-sensory interaction experience in different real scenes.

[0074] In a further embodiment, when the change in the scene feature parameters exceeds a preset threshold, the regeneration and synchronous update of the multi-sensory feedback parameters are triggered.

[0075] The system pre-establishes a user sensory preference parameter configuration module on the terminal side to store sensory preference information related to user habits. The user sensory preference parameters include at least one or more preference settings for visual feedback, auditory feedback, and tactile feedback, such as feedback intensity preference, feedback sensitivity preference, and feedback presentation method preference.

[0076] When generating a multi-sensory interactive feedback scheme, the interactive feedback generation module, based on the basic feedback strategy determined according to the scene type and scene details, further incorporates the user sensory preference parameters to personalize the basic feedback strategy. Specifically, for the same scene type and scene details, different users may have significantly different suitable feedback methods and intensity due to differences in perception abilities and usage habits. By introducing user sensory preference parameters, the problem of poor user experience caused by using a uniform feedback scheme can be avoided.

[0077] For example, when a user sets a lower preference for tactile feedback intensity in the user sensory preference parameters, the system will reduce the vibration intensity or duration accordingly, even in scenarios where tactile cues are required, and compensate through visual or auditory feedback to ensure that the overall multi-sensory feedback effect remains complete and coordinated.

[0078] Through the above methods, this implementation scheme enables the multi-sensory linkage feedback scheme to not only meet the requirements of scene adaptation, but also further reflect its ability to adapt to individual user differences, thereby improving the acceptability of the multi-sensory interaction experience and the comfort of long-term use.

[0079] In a further embodiment, when generating the multi-sensory feedback parameters, the multi-sensory feedback parameters are also adjusted in a personalized manner by incorporating sensory preference parameters pre-configured by the user.

[0080] During the execution of multi-sensory interactive feedback, the system continuously monitors changes in the characteristics of the real-world scene. When a change in the real-world scene is detected, and this change causes the original multi-sensory interactive feedback scheme to no longer match the current scene, the system triggers a dynamic adjustment process.

[0081] Specifically, the dynamic adjustment process includes reacquiring the updated scene feature parameters, performing a fusion analysis on the updated scene feature parameters, and modifying the current multi-sensory linkage feedback scheme based on the updated scene type and scene details. The modification may include gradual adjustments to the feedback parameters, or switching to a new multi-sensory linkage feedback scheme when necessary.

[0082] When making dynamic adjustments, the system not only considers changes in the current scene characteristics but also comprehensively takes into account user sensory preference parameters to avoid frequent and significant feedback changes from interfering with the user. For example, as ambient noise gradually increases, the system can gradually enhance visual feedback rather than abruptly changing the feedback method, making the feedback adjustment process smoother and more natural.

[0083] Through the aforementioned dynamic adjustment mechanism, this implementation method enables multi-sensory interactive feedback to maintain stability and continuity in the face of continuous changes in the real-world scenario, avoiding abrupt feedback or experience interruption caused by scene changes, thereby further improving the overall quality of multi-sensory interactive feedback.

[0084] Another aspect of this invention provides a multi-sensory interactive feedback system for mobile terminals based on real-world scenarios. Please refer to [link to relevant documentation]. Figure 3 ,include: Scene feature acquisition module 1 is used to acquire multi-dimensional scene feature parameters; Scene recognition and analysis module 2 is used to determine the scene type and scene detail parameters; Multi-sensory linkage control module 3 is used to generate multi-sensory feedback parameters; Sensory feedback execution module 4 is used to synchronously execute multi-sensory linkage feedback; The modules are configured to work together to execute the method described above.

[0085] The scene feature acquisition module 1 is used to acquire scene feature parameters related to the real environment from multiple sensors of the mobile terminal. These scene feature parameters include at least one or more of the following: ambient lighting information, ambient noise information, terminal motion state information, and location information. This module provides the system with raw data input reflecting the current state of the real scene through the unified acquisition of data from different sensors.

[0086] The scene recognition and analysis module 2 is connected to the scene feature acquisition module 1 and is used to preprocess and fuse the acquired multi-dimensional scene feature parameters. Specifically, the scene recognition and analysis module 2 performs noise reduction, time synchronization, and scale unification on various scene feature parameters, and generates fused scene feature results based on a preset fusion strategy, thereby providing a stable and reliable data foundation for subsequent scene recognition. Simultaneously, it can identify the scene type and scene details corresponding to the current real-world scene based on the fused scene feature results. By analyzing the correlation between multi-dimensional scene features, the scene recognition and analysis module 2 achieves hierarchical recognition of the real-world environment, enabling the system to distinguish different scene categories and further characterize the specific state differences within the same scene category.

[0087] The multi-sensory linkage control module 3 is connected to the scene recognition and analysis module 2, and is used to generate a multi-sensory linkage feedback scheme based on the recognized scene type and scene details. This module pre-stores feedback strategy configurations corresponding to different scene types and scene details, and when generating the feedback scheme, it combines user sensory preference parameters provided by the user preference management module to personalize the feedback strategy, thereby forming a multi-sensory linkage feedback scheme that matches the current real scene and user habits.

[0088] The sensory feedback execution module 4 is connected to the multi-sensory linkage control module 3 and is used to analyze and execute the generated multi-sensory linkage feedback scheme. This module performs unified scheduling and coordinated control of the visual feedback unit, auditory feedback unit, and tactile feedback unit according to the temporal and intensity relationships of various sensory feedbacks in the feedback scheme, ensuring that the multiple sensory feedbacks remain coordinated and consistent during the output process.

[0089] In addition, the user preference management module stores and manages sensory preference parameters related to user habits, and provides support to the linkage feedback generation module and the multi-sensory feedback control module during the generation and dynamic adjustment of multi-sensory linkage feedback schemes, so that the system can continuously adapt to the individual differences of different users during long-term use.

[0090] Through the coordinated operation of the above system modules, this implementation method can fully realize a multi-sensory linkage feedback method based on real-world scenarios at the system architecture level, making the division of labor among functional modules clear, the data flow clear, and possessing good engineering feasibility and scalability.

[0091] In another aspect, the present invention provides a mobile terminal, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the mobile terminal performs the method described above.

[0092] The terminal device can be a smartphone, tablet computer, or other mobile terminal device with multi-sensory interaction capabilities. It includes at least a processor, memory, sensor components, and multi-sensory feedback components. Each component is connected and interacts with data through an internal bus or communication interface of the terminal.

[0093] The processor is used to execute program instructions stored in the memory to implement the functions described in the foregoing embodiments, such as scene feature acquisition, scene analysis, multi-sensory linkage feedback generation, and control. The processor can be a general-purpose processor, a special-purpose processor, or a combination of both, and its specific form does not constitute a limitation of the present invention.

[0094] The memory is used to store data such as program instructions, configuration parameters related to scene recognition, and user sensory preference parameters. When the program instructions are executed by the processor, the terminal device can operate according to the method steps described above and implement each functional module described above at the system level.

[0095] The sensor component is used to collect multi-dimensional scene feature information related to the real scene. The sensor component includes at least one or more of a light sensor, a microphone, an acceleration sensor, a gyroscope, and a positioning module. Through the sensor component, the terminal device can obtain scene feature parameters reflecting environmental light, environmental noise, device motion state, and position information, providing basic data for subsequent scene recognition and linkage feedback.

[0096] The multi-sensory feedback component is used to output multi-sensory linkage feedback to the user. The multi-sensory feedback component includes at least one or more of a display screen, a speaker, and a vibration unit. Under the control of the processor, the multi-sensory feedback component outputs corresponding visual feedback, auditory feedback, and tactile feedback according to the generated multi-sensory linkage feedback scheme.

[0097] In the specific operation process, the processor first calls the sensor component to collect real-scene feature information, and processes and analyzes the collected scene feature information based on the program instructions stored in the memory to identify the scene type and scene details corresponding to the current scene; subsequently, the processor generates a target multi-sensory linkage feedback scheme according to the recognition result and user sensory preference parameters, and controls the multi-sensory feedback component to perform collaborative output according to the target multi-sensory linkage feedback scheme.

[0098] When the terminal device detects a change in the real scene, the processor can dynamically adjust the current multi-sensory linkage feedback scheme based on the updated scene feature information, so that the multi-sensory feedback output by the terminal device continuously matches the real scene.

[0099] Through the above structure and operation mode of the terminal device, this embodiment can completely implement the multi-sensory linkage feedback method based on the real scene under the existing hardware conditions, and has good engineering feasibility and practical application value.

[0100] In summary, the present invention focuses on the multi-sensory interaction problem of mobile terminals in real scenes, and constructs a complete, coherent, and engineering-realizable technical solution from scene feature collection, scene analysis and recognition, multi-sensory linkage feedback generation to feedback output execution. By uniformly processing and comprehensively analyzing multi-dimensional scene features at the system level, the terminal can form a stable and accurate scene perception of the real environment, thereby providing a reliable basis for subsequent multi-sensory linkage feedback.

[0101] Building upon this foundation, the present invention does not simply superimpose visual, auditory, and tactile feedback. Instead, it focuses on scene type and details, combined with user sensory preference parameters, to coordinate and dynamically regulate multiple sensory feedbacks holistically. This ensures consistency and coherence in the timing, intensity, and presentation of different sensory feedbacks. This approach effectively avoids the problems of fragmented multi-sensory feedback and inconsistent experiences found in existing technologies.

[0102] Meanwhile, by introducing a dynamic adjustment mechanism, the system can promptly update or correct the multi-sensory interactive feedback scheme when the real-world scenario changes, ensuring that the feedback output from the terminal continuously adapts to the current environmental state, further enhancing the stability and adaptability of the multi-sensory interactive experience. All of the above technical solutions can be implemented under existing mobile terminal hardware conditions, possessing clear implementation paths and good engineering feasibility.

[0103] Therefore, the multi-sensory linkage feedback method and system based on real-world scenarios proposed in this invention have significant technical effects in improving the consistency of multi-sensory interaction and scene adaptability of mobile terminals. They are applicable to a variety of practical application scenarios and have good practical value and promotion prospects.

[0104] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Images