Interactive image browsing method and device based on multiple sensors

Abstract

The invention discloses an interactive image browsing method based on multiple sensors. And the mobile terminal senses any one of plane movement, vertical movement or rotation in the pitch angle direction of the mobile terminal through the inertial measurement unit and / or the camera. The mobile terminal calculates an X-axis offset according to the X-axis displacement; calculating a Y-axis offset according to the Y-axis displacement; the image is moved. And the mobile terminal calculates a zooming multiple according to any one of the Z-axis displacement or the pitch angle variation and carries out zooming. Once the mobile terminal detects that the state locking key is pressed down or detects a touch gesture of state locking, the mobile terminal is in a locking state; and once it is detected that the state unlocking key is pressed down or a state unlocking touch gesture is detected, the state unlocking device is in an unlocking state. According to the method, one-time long-distance movement is decomposed into multiple-time short-distance movement, one-time large-angle rotation is decomposed into multiple-time small-angle rotation, and long-distance movement of the display area and large-scale adjustment of the zooming multiple are facilitated.

Description

Technical Field

[0001] This invention relates to a method for browsing electronic images and electronic maps on a mobile terminal. Background Technology

[0002] Browsing electronic images on mobile devices such as smartphones, tablets, PDAs (personal digital assistants), cameras, and handheld game consoles is a common operation, and many touch gestures are already available for this purpose. For example, a single-finger swipe adjusts the display area of ​​an electronic image on the mobile device screen, while pinching or spreading two fingers zooms in or out. Electronic maps can be considered as ultra-large or infinitely large electronic images, and browsing electronic maps is essentially no different from browsing electronic images.

[0003] When users need to view a specific area in an electronic image, they often need to repeatedly swipe with one finger to adjust the displayed area of ​​the image on the mobile device screen. When users need to view certain details in an electronic image, they often need to repeatedly pinch or swipe with two fingers to zoom in and out. In outdoor or mobile scenarios, users typically hold the mobile device with one hand and use the other hand to browse electronic images, thus occupying both hands. Summary of the Invention

[0004] The technical problem to be solved by this invention is: how to enable users to browse electronic images or electronic maps using only one hand on a mobile terminal, and to facilitate long-distance movement of the display area and significant adjustment of the zoom level.

[0005] To address the aforementioned technical problems, this invention proposes a method for interactive image browsing based on multiple sensors, comprising the following steps: Step S1: When a mobile terminal displays an electronic image or electronic map on the screen, the mobile terminal uses an inertial measurement unit and / or a camera to perceive its planar movement in real time, and also perceives any one of its vertical movement or rotation in the pitch direction; it acquires in real time the X-axis and Y-axis displacements of the mobile terminal on the horizontal plane, and also acquires in real time any one of the Z-axis displacements or pitch angle changes of the mobile terminal on the vertical line; subsequently, it proceeds simultaneously to steps S2a and S2b. Step S2a: The mobile terminal calculates the X-axis offset between the target display area and the current display area of ​​the electronic image or electronic map in real time based on the real-time acquired X-axis displacement; the mobile terminal calculates the Y-axis offset between the target display area and the current display area of ​​the electronic image or electronic map in real time based on the real-time acquired Y-axis displacement; the mobile terminal moves accordingly based on the X-axis and Y-axis offsets between the target display area and the current display area of ​​the electronic image or electronic map, adjusting the current display area on the screen to the target display area; then proceeds to step S3. Step S2b: The mobile terminal calculates the scaling factor between the target display area and the current display area of ​​the electronic image or electronic map in real time based on either the real-time acquired Z-axis displacement or the pitch angle change; the mobile terminal adjusts the current display area on the screen to the target display area based on the scaling factor between the target display area and the current display area of ​​the electronic image or electronic map; then proceeds to step S3. Step S3: Once the mobile terminal detects that the state lock button is pressed or a state lock touch gesture is detected, it is in a locked state and stops executing steps S1, S2a, and S2b; once the mobile terminal detects that the state unlock button is pressed or a state unlock touch gesture is detected, it is in an unlocked state and resumes executing steps S1, S2a, and S2b based on the current display area and zoom level of the electronic image or electronic map on the mobile terminal screen.

[0006] Furthermore, in step S1, a rectangular coordinate system is formed with the center of gravity of the mobile terminal as the origin O, the projection of the line connecting the smart terminal from left to right on the horizontal plane when the user holds the smart terminal as the X-axis, the projection of the line connecting the smart terminal from top to bottom on the horizontal plane when the user holds the smart terminal as the Y-axis, and the vertical line as the Z-axis.

[0007] Furthermore, in step S1, the inertial measurement unit includes a three-axis gyroscope and accelerometers in three directions, or a portion thereof, used to measure the angular velocity and acceleration of the object; the camera perceives the planar movement of the mobile terminal in real time by capturing changes in the characteristics of the surrounding environment in real time, perceives the vertical movement of the mobile terminal in real time by identifying the relative distance with the surrounding plane in real time, and perceives the rotation of the mobile terminal in the pitch direction in real time by identifying the relative angle with the surrounding plane in real time.

[0008] Furthermore, in step S2a, when the user holds the mobile terminal and moves it in any direction on the horizontal plane, the electronic image or electronic map on the screen will also change its display area along the same direction.

[0009] Further, in step S2b, when the mobile terminal moves vertically upward, it indicates that the electronic image or electronic map on the screen is reduced; at this time, the scaling factor is a positive reduction factor, indicating that the target display area of ​​the electronic image or electronic map is larger in display range but less detailed compared to the current display area, and the value of the reduction factor is proportional to the absolute value of the Z-axis displacement; when the mobile terminal moves vertically downward, it indicates that the electronic image or electronic map on the screen is enlarged; at this time, the scaling factor is a positive magnification factor, indicating that the target display area of ​​the electronic image or electronic map is smaller in display range but more detailed compared to the current display area, and the value of the magnification factor is proportional to the absolute value of the Z-axis displacement. Alternatively, when the mobile terminal rotates in the pitch direction, causing the pitch angle to increase, it indicates that the electronic image or map on the screen is reduced; in this case, the scaling factor is a positive reduction factor, and the value of the reduction factor is proportional to the absolute value of the change in pitch angle. When the mobile terminal rotates in the pitch direction, causing the pitch angle to decrease, it indicates that the electronic image or map on the screen is enlarged; in this case, the scaling factor is a positive magnification factor, and the value of the magnification factor is proportional to the absolute value of the change in pitch angle.

[0010] Further, in step S3, the state lock button is a combination of one or more physical buttons on the mobile terminal being pressed simultaneously, and the state unlock button is a combination of one or more physical buttons on the mobile terminal being pressed simultaneously, and the state lock button is different from the state unlock button. Alternatively, the state lock button and the state unlock button are the same, and are called the state lock unlock button; in this case, the mobile terminal is in an unlocked state before it first detects that the state lock unlock button has been pressed; once the mobile terminal in the unlocked state detects that the state lock unlock button has been pressed, it enters a locked state and stops executing steps S1, S2a, and S2b; once the mobile terminal in the locked state detects that the state lock unlock button has been pressed, it enters an unlocked state and resumes executing steps S1, S2a, and S2b.

[0011] Further, in step S3, the touch gesture for state locking is different from the touch gesture for state unlocking. Alternatively, the touch gesture for state locking and the touch gesture for state unlocking are the same, referred to as the state lock unlock touch gesture; in this case, the mobile terminal is in an unlocked state before it first detects the state lock unlock touch gesture; once the mobile terminal in the unlocked state detects the state lock unlock touch gesture, it enters a locked state and stops executing steps S1, S2a, and S2b; once the mobile terminal in the locked state detects the state lock unlock touch gesture, it enters an unlocked state and resumes executing steps S1, S2a, and S2b.

[0012] Furthermore, in step S1, the external parameters of the inertial measurement unit and the camera are calibrated and the timestamps are aligned to ensure that the data acquired by the two types of sensors can be fused under a unified time axis; using the acceleration and angular velocity data acquired by the inertial measurement unit, the short-time displacement measurement and short-time pitch angle change measurement of the mobile terminal are performed independently; feature points of environmental features are extracted on adjacent image frames acquired by the camera, and the change of feature points between adjacent image frames is calculated to obtain the pose change of the mobile terminal; during long-term motion prediction, the pose change of the mobile terminal acquired by the camera is used to correct the drift error of the inertial measurement unit; when the mobile terminal is stationary or locked, zero bias reestimation and correction are performed to eliminate long-term accumulated errors.

[0013] Furthermore, in step S3, once the mobile terminal detects that the reset button has been pressed or a reset touch gesture has been detected, it restores the currently displayed electronic image or electronic map to the default display area and default zoom level, and sets the mobile terminal to the unlocked state.

[0014] This invention also proposes a multi-sensor-based interactive image browsing device, including a measurement unit, a mobile display unit, a zoom adjustment unit, and a lock / unlock unit. The measurement unit is used to sense the planar movement of the mobile terminal in real time when the mobile terminal displays an electronic image or electronic map on the screen, and also to sense any one of the mobile terminal's vertical movement or rotation in the pitch direction; it also acquires in real time the X-axis and Y-axis displacements of the mobile terminal on the horizontal plane, and any one of the Z-axis displacements or pitch angle changes of the mobile terminal on the vertical line. The mobile display unit is used to calculate the X-axis offset between the target display area and the current display area of ​​the electronic image or electronic map in real time based on the real-time acquired X-axis displacement of the mobile terminal; it is also used to calculate the Y-axis offset between the target display area and the current display area of ​​the electronic image or electronic map in real time based on the real-time acquired Y-axis displacement of the mobile terminal; and it is also used to move the display area of ​​the electronic image or electronic map accordingly based on the X-axis and Y-axis offsets, adjusting the current display area on the screen to the target display area. The scaling adjustment unit is used to calculate in real time the scaling factor between the target display area and the current display area of ​​the electronic image or electronic map based on either the real-time Z-axis displacement or the pitch angle change of the mobile terminal; it is also used to scale the electronic image or electronic map accordingly based on the scaling factor, adjusting the current display area on the screen to the target display area. The lock / unlock unit is used to lock the mobile terminal when a lock button is pressed or a lock-state touch gesture is detected, at which point the measurement unit, mobile display unit, and scaling adjustment unit all cease operation; it is also used to unlock the mobile terminal when an unlock button is pressed or an unlock-state touch gesture is detected, at which point the measurement unit, mobile display unit, and scaling adjustment unit all resume operation.

[0015] The technical effects achieved by this invention are: it provides users with a simple and convenient way to browse and control images, which can be operated with one hand; by decomposing a long-distance movement into multiple short-distance movements and a large-angle rotation into multiple small-angle rotations, it is very convenient to move the display area over long distances and adjust the zoom level significantly. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the interactive image browsing method based on multiple sensors proposed in this invention.

[0017] Figure 2 This is a schematic diagram of the structure of the interactive image browsing device based on multiple sensors proposed in this invention.

[0018] The attached diagram is labeled as follows: Measurement unit 1, Moving display unit 2, Zooming adjustment unit 3, Locking / unlocking unit 4. Detailed Implementation

[0019] Please see Figure 1 The method for interactive image browsing based on multiple sensors proposed in this invention includes the following steps.

[0020] Step S1: When the mobile terminal displays an electronic image or map on the screen, the mobile terminal uses an inertial measurement unit and / or a camera to perceive its planar movement in real time, as well as any one of its vertical movement or rotation in the pitch direction. It also acquires the X-axis and Y-axis displacements of the mobile terminal on a plane parallel to the ground (horizontal plane) in real time, and any one of its Z-axis displacement or pitch angle change in a direction perpendicular to the ground (vertical line). Then, steps S2a and S2b are executed simultaneously. This invention uses a rectangular coordinate system with the mobile terminal's center of gravity as the origin O, the projection of the line connecting the smart terminal from left to right on the horizontal plane as the X-axis, the projection of the line connecting the smart terminal from top to bottom on the horizontal plane as the Y-axis, and the vertical line as the Z-axis. It is important to note that the meanings of the X-axis and Y-axis are exactly opposite when the mobile terminal is used in portrait and landscape modes, depending on the direction in which the user is holding the smart terminal.

[0021] Planar movement of a mobile terminal refers to its translational motion on a horizontal plane—such as left-right or up-down movement—which generates X-axis and / or Y-axis displacement. Vertical movement of a mobile terminal refers to its movement along a vertical line, which generates Z-axis displacement. Rotation of a mobile terminal can be decomposed into azimuth, pitch, and roll. This invention focuses only on the rotation of the mobile terminal in the pitch direction, which refers to the rotation of the smart terminal in the YOZ plane when held by the user, resulting in a change in pitch angle. For example, when the user holds the smart terminal, if the top of the smart terminal is above the horizontal plane, the pitch angle is positive; if the top of the smart terminal is on the horizontal plane, the pitch angle is zero; and if the top of the smart terminal is below the horizontal plane, the pitch angle is negative. The motion of a mobile terminal in a real environment is often complex, combining planar movement, vertical movement, and rotation. By decomposing these elements, the physical quantities of interest in this invention can be obtained.

[0022] The inertial measurement unit (IMU) typically includes a three-axis gyroscope and three accelerometers, or a subset thereof, to measure the angular velocity and acceleration of an object, thus enabling real-time sensing of the mobile terminal's motion, attitude, direction, and angle. The camera senses the mobile terminal's planar movement in real time by capturing changes in the surrounding environment, its vertical movement in real time by identifying its relative distance to the surrounding plane, and its rotation in the pitch direction in real time by identifying its relative angle to the surrounding plane. Therefore, either the inertial measurement unit or the camera can be used alone to acquire the physical quantities of interest in this invention.

[0023] Step S2a: The mobile terminal calculates the X-axis offset between the target display area and the current display area of ​​the electronic image or map in real time based on the real-time acquired X-axis displacement. The mobile terminal calculates the Y-axis offset between the target display area and the current display area of ​​the electronic image or map in real time based on the real-time acquired Y-axis displacement. The mobile terminal moves accordingly based on the X-axis and Y-axis offsets between the target display area and the current display area of ​​the electronic image or map, adjusting the current display area on the screen to the target display area. Then proceed to step S3.

[0024] When a user holds a mobile terminal and moves it horizontally in any direction, the electronic image or map on the screen will change its display area along the same direction. The ratio of the X-axis offset to the X-axis displacement can be a default first movement parameter or can be set by the user. The ratio of the Y-axis offset to the Y-axis displacement can be a default second movement parameter or can be set by the user. The first and second movement parameters can be the same or different. Compared to changing the display area of ​​an electronic image or map on the mobile terminal screen by swiping with one finger, the interaction method of changing the display area of ​​an electronic image or map on the mobile terminal screen by moving the mobile terminal is more intuitive and especially suitable for one-handed operation, as the other hand does not need to touch the mobile terminal.

[0025] Step S2b: The mobile terminal calculates the scaling factor between the target display area and the current display area of ​​the electronic image or electronic map based on either the real-time acquired Z-axis displacement or pitch angle. The mobile terminal then adjusts the current display area on the screen to match the target display area based on this scaling factor. The process then proceeds to step S3.

[0026] As a first example, when the mobile terminal moves vertically upwards, it indicates that the electronic image or map on the screen is being reduced in size, similar to a magnifying glass moving away from paper. At this time, the Z-axis displacement is, for example, a positive number, and the scaling factor is a positive reduction factor, indicating that the target display area of ​​the electronic image or map becomes larger but less detailed compared to the current display area, and the value of the reduction factor is proportional to the absolute value of the Z-axis displacement.

[0027] As a second example, when the mobile terminal moves vertically downwards, it represents magnifying the electronic image or map on the screen, similar to a magnifying glass approaching a piece of paper. In this case, the Z-axis displacement is, for example, negative, while the scaling factor is a positive magnification factor, indicating that the target display area of ​​the electronic image or map becomes smaller but more detailed compared to the current display area, and the value of the magnification factor is proportional to the absolute value of the Z-axis displacement.

[0028] As a third example, when the mobile terminal rotates in the pitch direction, causing the pitch angle to increase (including changing from a negative to a positive pitch angle), it indicates that the electronic image or electronic map on the screen is reduced in size. In this case, the change in pitch angle is, for example, a positive number, and the scaling factor is a positive scaling factor, and the value of the scaling factor is proportional to the absolute value of the change in pitch angle.

[0029] As a fourth example, when the rotation of the mobile terminal in the pitch direction causes the pitch angle to decrease (including a change from a positive to a negative pitch angle), it indicates magnification of the electronic image or map on the screen. In this case, the change in pitch angle is, for example, negative, and the scaling factor is a positive magnification factor, and the value of the magnification factor is proportional to the absolute value of the change in pitch angle.

[0030] As the user holds the mobile device and moves it closer to their eye, the electronic image or map on the screen is zoomed out; moving it further away zooms it out, similar to using a magnifying glass. The ratio of the zoom level to the absolute value of the Z-axis displacement can be a default third movement parameter or a user-defined parameter. Alternatively, raising the top of the mobile device zooms out and lowering it zooms it out. The ratio of the zoom level to the absolute value of the pitch angle change can be a default fourth movement parameter or a user-defined parameter. This interactive method of vertically moving the mobile device or changing its pitch angle is more intuitive than pinching or extending two fingers, especially suitable for one-handed operation, as the other hand does not need to touch the mobile device. Furthermore, users can achieve progressive zooming from a macro overview to micro details by continuously raising or lowering the top of the mobile device, such as gradually zooming from a full map view to street-level details.

[0031] Step S3: Once the mobile terminal detects that the state lock button has been pressed or a state lock touch gesture has been detected, it is in a locked state and stops executing steps S1, S2a, and S2b. Once the mobile terminal detects that the state unlock button has been pressed or a state unlock touch gesture has been detected, it is in an unlocked state and resumes executing steps S1, S2a, and S2b based on the current display area and zoom level of the electronic image or electronic map on the mobile terminal's screen.

[0032] As a first example, the state lock button is a combination of one or more physical buttons on the mobile terminal being pressed simultaneously, and the state unlock button is a combination of one or more physical buttons on the mobile terminal being pressed simultaneously, and the state lock button is different from the state unlock button.

[0033] As a second example, the state-locked touch gesture is different from the state-unlocked touch gesture.

[0034] As a third example, the state lock button and the state unlock button are the same, referred to as the state lock unlock button. In this case, the mobile terminal is in an unlocked state before it first detects that the state lock unlock button has been pressed. Once the mobile terminal in the unlocked state detects that the state lock unlock button has been pressed, it enters a locked state and stops executing steps S1, S2a, and S2b. Once the mobile terminal in the locked state detects that the state lock unlock button has been pressed, it enters an unlocked state and resumes executing steps S1, S2a, and S2b.

[0035] As a fourth example, the touch gesture for locking the state is the same as the touch gesture for unlocking the state, called the state lock unlock touch gesture, such as a long press on the screen. In this case, the mobile terminal is in an unlocked state before it first detects the state lock unlock touch gesture. Once the unlocked mobile terminal detects the state lock unlock touch gesture, it enters a locked state and stops executing steps S1, S2a, and S2b. Once the locked mobile terminal detects the state lock unlock touch gesture, it enters an unlocked state and resumes executing steps S1, S2a, and S2b.

[0036] State locking and unlocking is a major technical innovation of this invention. Without state locking, when a user moves or rotates the mobile terminal to adjust the electronic image or map on the screen to the area of ​​interest and a suitable zoom level, the position and angle of the mobile terminal may not be suitable for viewing. If the user moves the mobile terminal to a more convenient viewing position, the previously adjusted display area and zoom level will change. To solve these problems, this invention adds state locking and unlocking. After the user moves or rotates the mobile terminal to adjust the electronic image or map on the screen to the area of ​​interest and a suitable zoom level, state locking is performed. At this time, the current display area and zoom level of the electronic image or map on the mobile terminal screen remain unchanged. Subsequently, the mobile terminal can be moved to the most convenient position in front of the user for viewing without worrying about the adjusted display area and zoom level of the electronic image or map changing.

[0037] Without state locking, even if a user moves their mobile device from the far left or far right, or from the far top or far bottom, they may still not reach the target area of ​​the electronic image or map, assuming their position remains unchanged. This can happen when browsing very large or high-resolution electronic images or maps, as the user's arm movement range is limited. To address this problem, this invention incorporates state locking and unlocking. For example, the user locks the state after moving the mobile device from the far left or far right, then unlocks it after moving it back to the far left, and so on, breaking down long-distance planar movement into multiple short-distance planar movements. This is particularly advantageous for users locating the target area from a distance when browsing very large or high-resolution electronic images or maps.

[0038] Without state locking, even if a user moves the mobile device from the nearest accessible end to the farthest end, or rotates the device within their comfort zone in terms of pitch, they may still not achieve the desired zoom level, especially when browsing extremely large or high-resolution electronic images or maps, because the user's arm's vertical movement range and wrist rotation angle are limited. To address these challenges, this invention incorporates state locking and unlocking. For example, after moving the mobile device from the nearest accessible end to the farthest end, the state is locked; then, after moving the device back to the nearest end, the state is unlocked; and so on, breaking down long-distance vertical movements into multiple short-distance vertical movements. Similarly, after rotating the mobile device from a comfortable position to its maximum pitch angle, the state is locked; then, after moving the device back to a comfortable position, the state is unlocked; and so on, breaking down large-angle rotations into multiple small-angle rotations. This is especially beneficial for users who want to significantly adjust the zoom level when browsing ultra-large or ultra-high resolution electronic images or maps.

[0039] In addition, step S3 can also be used for fine-grained browsing in stages and steps—first locking the large area, then fine-tuning the small area, improving the continuity of browsing operations.

[0040] In step S1, it is preferable to simultaneously use an inertial measurement unit (IMU) and a camera to perceive the mobile terminal's planar movement, vertical movement, and rotation in the pitch direction in real time. First, the external parameters of the IMU and camera are calibrated and timestamps are aligned to ensure that the data acquired by the two types of sensors can be fused along a unified time axis. Then, using the acceleration and angular velocity data acquired by the IMU, short-term displacement and short-term pitch angle changes of the mobile terminal are measured independently to obtain smooth, low-latency short-term motion predictions. The camera does not participate in short-term motion prediction. In this scenario, "short time" typically refers to less than 1000 milliseconds. When the independent use of the IMU exceeds this short time (typically greater than 1000 milliseconds), the IMU may drift due to integration accumulation. To address this problem, feature points of environmental features (such as desktop texture) are extracted from adjacent image frames acquired by the camera. The changes in feature points between adjacent image frames are calculated to obtain a more accurate change in the mobile terminal's pose, providing low-drift absolute motion observation. During long-term motion prediction, the pose changes of the mobile terminal acquired by the camera are used to correct the drift error of the inertial measurement unit (IMU). Visual observations of feature points on adjacent image frames are used to correct the displacement and pitch angle changes acquired by the IMU, achieving complementary fusion of the two types of sensors. Therefore, by combining the short-term stability of the IMU with the long-term accuracy of visual information, filtering or optimization methods are used to fuse the IMU measurement data and the visual information data acquired by the camera in real time, resulting in high-precision mobile terminal position and attitude. Finally, when the mobile terminal is stationary or locked, zero-bias reestimation and correction are performed to further eliminate long-term accumulated errors. Through this collaborative mechanism combining short-term prediction and long-term calibration, the drift of the IMU can be significantly suppressed while maintaining rapid response, achieving stable and accurate recognition of the mobile terminal's movement and rotation. This effectively overcomes the offset caused by a single sensor, ensuring the real-time performance and accuracy of interactive operations in image browsing scenes.

[0041] In step S3, once the mobile terminal detects that the reset button has been pressed or a reset touch gesture has been detected, it restores the currently displayed electronic image or map to the default display area and default zoom level, and sets the mobile terminal to the unlocked state. The reset button is one or a combination of multiple physical buttons pressed simultaneously on the mobile terminal. The default display area is, for example, the display area when the electronic image or map is first opened for viewing, the entire view of the electronic image or map, or the central area of ​​the electronic image or map. The default zoom level is, for example, the default zoom level when the electronic image or map is first opened for viewing, or the zoom level that allows the mobile terminal screen to display the entire view of the electronic image or map, or the zoom level that displays the electronic image or map at its original resolution (neither zooming in nor zooming out). The reset button or touch gesture allows the user to re-view the currently displayed electronic image or map in its initial state, or end the viewing of the currently displayed electronic image or map, or start viewing a new electronic image or map, improving the user experience and flexibility.

[0042] Please see Figure 2 The interactive image browsing device based on multiple sensors proposed in this invention includes a measurement unit 1, a mobile display unit 2, a zoom adjustment unit 3, and a lock / unlock unit 4. Figure 2 The device shown corresponds to Figure 1 The method shown.

[0043] The measurement unit 1 is used to sense the planar movement of the mobile terminal in real time when the mobile terminal displays electronic images or electronic maps on the screen, and also to sense any one of the vertical movement or rotation in the pitch direction of the mobile terminal in real time; it also acquires the X-axis and Y-axis displacement of the mobile terminal on the horizontal plane in real time, and also acquires any one of the Z-axis displacement or pitch angle change of the mobile terminal on the vertical line in real time. The measurement unit 1 is, for example, an inertial measurement unit and / or a camera built into the mobile terminal.

[0044] The mobile display unit 2 is used to calculate the X-axis offset between the target display area and the current display area of ​​the electronic image or electronic map in real time based on the real-time X-axis displacement of the mobile terminal; it is also used to calculate the Y-axis offset between the target display area and the current display area of ​​the electronic image or electronic map in real time based on the real-time Y-axis displacement of the mobile terminal; and it is also used to move the display area of ​​the electronic image or electronic map accordingly based on the X-axis offset and Y-axis offset, so as to adjust the current display area on the screen to the target display area.

[0045] The scaling adjustment unit 3 is used to calculate the scaling factor between the target display area and the current display area of ​​the electronic image or electronic map in real time based on either the Z-axis displacement or the pitch angle of the mobile terminal acquired in real time; it is also used to scale the electronic image or electronic map accordingly based on the scaling factor, and adjust the current display area on the screen to the target display area.

[0046] The locking / unlocking unit 4 is used to lock the mobile terminal when the state lock button is pressed or a state lock touch gesture is detected, at which time the measurement unit 1, the mobile display unit 2, and the scaling adjustment unit 3 all stop working; it is also used to unlock the mobile terminal when the state unlock button is pressed or a state unlock touch gesture is detected, at which time the measurement unit 1, the mobile display unit 2, and the scaling adjustment unit 3 all resume working.

[0047] Compared with existing technologies, the interactive image browsing method proposed in this invention has the following beneficial effects.

[0048] First, it offers a superior user experience. This invention provides users with an interactive experience similar to a "virtual magnifying glass" based on multimodal sensors (inertial measurement unit and camera), allowing for convenient one-handed operation.

[0049] Secondly, through unique locking and unlocking operations, this invention can decompose a single long-distance movement of a mobile terminal into multiple short-distance movements, and can also decompose a single large-angle rotation of a mobile terminal into multiple small-angle rotations. It is particularly suitable for moving the display area of ​​electronic images or electronic maps over long distances and significantly adjusting the zoom level of electronic images or electronic maps, thereby further improving the user's operating experience.

[0050] Third, the present invention can selectively fuse measurement results from multiple sensors, overcoming the limitations of single-sensor measurement and improving robustness.

[0051] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for interactive image browsing based on multiple sensors, characterized in that, Includes the following steps; Step S1: When the mobile terminal displays an electronic image or electronic map on the screen, the mobile terminal uses an inertial measurement unit and / or camera to perceive the planar movement of the mobile terminal in real time, and also perceives any one of the vertical movement or rotation in the pitch direction of the mobile terminal in real time; it acquires the X-axis displacement and Y-axis displacement of the mobile terminal on the horizontal plane in real time, and also acquires any one of the Z-axis displacement or pitch angle change of the mobile terminal on the vertical line in real time; then proceed to steps S2a and S2b simultaneously. Step S2a: The mobile terminal calculates the X-axis offset between the target display area and the current display area of ​​the electronic image or electronic map in real time based on the real-time acquired X-axis displacement. The mobile terminal calculates the Y-axis offset between the target display area and the current display area of ​​the electronic image or electronic map in real time based on the real-time acquired Y-axis displacement. The mobile terminal moves accordingly based on the X-axis and Y-axis offsets between the target display area and the current display area of ​​the electronic image or electronic map, adjusting the current display area on the screen to the target display area; Then proceed to step S3; Step S2b: The mobile terminal calculates the scaling factor between the target display area of ​​the electronic image or electronic map and the current display area in real time based on either the real-time Z-axis displacement or the pitch angle change. The mobile terminal adjusts the current display area on the screen to the target display area based on the scaling factor between the target display area and the current display area of ​​the electronic image or electronic map. Then proceed to step S3; Step S3: Once the mobile terminal detects that the state lock button is pressed or a state lock touch gesture is detected, it is in a locked state and stops executing steps S1, S2a, and S2b; once the mobile terminal detects that the state unlock button is pressed or a state unlock touch gesture is detected, it is in an unlocked state and resumes executing steps S1, S2a, and S2b based on the current display area and zoom level of the electronic image or electronic map on the mobile terminal screen.

2. The method for interactive image browsing based on multiple sensors according to claim 1, characterized in that, In step S1, a rectangular coordinate system is formed with the center of gravity of the mobile terminal as the origin O, the projection of the line connecting the smart terminal from left to right on the horizontal plane when the user holds the smart terminal as the X-axis, the projection of the line connecting the smart terminal from top to bottom on the horizontal plane when the user holds the smart terminal as the Y-axis, and the vertical line as the Z-axis.

3. The method for interactive image browsing based on multiple sensors according to claim 1, characterized in that, In step S1, the inertial measurement unit includes a three-axis gyroscope and accelerometers in three directions, or a portion thereof, used to measure the angular velocity and acceleration of an object; the camera perceives the planar movement of the mobile terminal in real time by capturing changes in the characteristics of the surrounding environment, perceives the vertical movement of the mobile terminal in real time by identifying the relative distance to the surrounding plane, and perceives the rotation of the mobile terminal in the pitch direction in real time by identifying the relative angle to the surrounding plane.

4. The method for interactive image browsing based on multiple sensors according to claim 1, characterized in that, In step S2a, the user holds the mobile terminal and moves it in any direction on the horizontal plane. The electronic image or electronic map on the screen will also change its display area along the same direction.

5. The method for interactive image browsing based on multiple sensors according to claim 1, characterized in that, In step S2b, when the mobile terminal moves vertically upward, it indicates that the electronic image or electronic map on the screen is reduced; the scaling factor at this time is a positive reduction factor, indicating that the target display area of ​​the electronic image or electronic map is larger than the current display area but has fewer details, and the value of the reduction factor is proportional to the absolute value of the Z-axis displacement; when the mobile terminal moves vertically downward, it indicates that the electronic image or electronic map on the screen is enlarged; the scaling factor at this time is a positive magnification factor, indicating that the target display area of ​​the electronic image or electronic map is smaller than the current display area but has more details, and the value of the magnification factor is proportional to the absolute value of the Z-axis displacement. Alternatively, when the mobile terminal rotates in the pitch direction, causing the pitch angle to increase, it indicates that the electronic image or map on the screen is reduced; in this case, the scaling factor is a positive reduction factor, and the value of the reduction factor is proportional to the absolute value of the change in pitch angle. When the mobile terminal rotates in the pitch direction, causing the pitch angle to decrease, it indicates that the electronic image or map on the screen is enlarged; in this case, the scaling factor is a positive magnification factor, and the value of the magnification factor is proportional to the absolute value of the change in pitch angle.

6. The method for interactive image browsing based on multiple sensors according to claim 1, characterized in that, In step S3, the state lock button is a combination of one or more physical buttons on the mobile terminal being pressed simultaneously, and the state unlock button is a combination of one or more physical buttons on the mobile terminal being pressed simultaneously. The state lock button is different from the state unlock button. Alternatively, the status lock button and the status unlock button are the same, and it is called the status lock unlock button; At this time, before the mobile terminal first detects that the state lock unlock button has been pressed, the mobile terminal is in the unlocked state; once the mobile terminal in the unlocked state detects that the state lock unlock button has been pressed, it is in the locked state and stops executing steps S1, S2a and S2b; once the mobile terminal in the locked state detects that the state lock unlock button has been pressed, it is in the unlocked state and resumes executing steps S1, S2a and S2b.

7. The method for interactive image browsing based on multiple sensors according to claim 1, characterized in that, In step S3, the touch gesture for state locking is different from the touch gesture for state unlocking. Alternatively, the touch gesture for state lock and the touch gesture for state unlock are the same, which is called state lock unlock touch gesture; At this time, before the mobile terminal first detects the state lock unlock touch gesture, the mobile terminal is in the unlocked state; once the mobile terminal in the unlocked state detects the state lock unlock touch gesture, it is in the locked state and stops executing steps S1, S2a and S2b; once the mobile terminal in the locked state detects the state lock unlock touch gesture, it is in the unlocked state and resumes executing steps S1, S2a and S2b.

8. The method for interactive image browsing based on multiple sensors according to claim 1, characterized in that, In step S1, the external parameters of the inertial measurement unit (IMU) and the camera are calibrated and the timestamps are aligned to ensure that the data acquired by the two types of sensors can be fused under a unified time axis. Using the acceleration and angular velocity data acquired by the IMU, short-term displacement and short-term pitch angle changes of the mobile terminal are measured independently. Feature points of environmental features are extracted from adjacent image frames acquired by the camera, and the changes in feature points between adjacent image frames are calculated to obtain the pose change of the mobile terminal. During long-term motion prediction, the pose change of the mobile terminal acquired by the camera is used to correct the drift error of the IMU. Zero-bias reestimation and correction are performed when the mobile terminal is stationary or locked to eliminate long-term accumulated errors.

9. The method for interactive image browsing based on multiple sensors according to claim 1, characterized in that, In step S3, once the mobile terminal detects that the reset button has been pressed or a reset touch gesture has been detected, it restores the currently displayed electronic image or electronic map to the default display area and default zoom level, and sets the mobile terminal to the unlocked state.

10. A device for interactive image browsing based on multiple sensors, characterized in that, Includes a measurement unit, a mobile display unit, a zoom adjustment unit, and a lock / unlock unit; The measurement unit is used to sense the planar movement of the mobile terminal in real time when the mobile terminal displays electronic images or electronic maps on the screen, and also to sense any one of the vertical movement or rotation in the pitch direction of the mobile terminal in real time; and to obtain the X-axis displacement and Y-axis displacement of the mobile terminal on the horizontal plane in real time, and also to obtain any one of the Z-axis displacement or pitch angle change of the mobile terminal on the vertical line in real time. The mobile display unit is used to calculate the X-axis offset between the target display area and the current display area of ​​the electronic image or electronic map in real time based on the real-time X-axis displacement of the mobile terminal; it is also used to calculate the Y-axis offset between the target display area and the current display area of ​​the electronic image or electronic map in real time based on the real-time Y-axis displacement of the mobile terminal; and it is also used to move the display area of ​​the electronic image or electronic map accordingly based on the X-axis offset and Y-axis offset, so as to adjust the current display area on the screen to the target display area. The scaling adjustment unit is used to calculate the scaling factor of the target display area of ​​the electronic image or electronic map compared with the current display area in real time, based on either the Z-axis displacement or the pitch angle change of the mobile terminal acquired in real time. It is also used to scale electronic images or electronic maps according to the scaling factor, and adjust the current display area on the screen to the target display area; The lock / unlock unit is used to lock the mobile terminal when the state lock button is detected to be pressed or a state lock touch gesture is detected, at which time the measurement unit, the mobile display unit, and the scaling adjustment unit all stop working; it is also used to unlock the mobile terminal when the state unlock button is detected to be pressed or a state unlock touch gesture is detected, at which time the measurement unit, the mobile display unit, and the scaling adjustment unit all resume working.

Images