Interaction method and device, electronic equipment and storage medium

Abstract

The invention discloses an interaction method and device and electronic equipment, and belongs to the technical field of electronic equipment. The method comprises the following steps: acquiring first optical flow information of continuous image frames captured by a camera under the condition that the camera detects that a target object enters an optical path area of an auxiliary light source; and executing a corresponding operation on a current display interface based on the first optical flow information.

Description

Technical Field

[0001] This application belongs to the field of electronic equipment technology, specifically relating to an interaction method, device, and electronic equipment. Background Technology

[0002] Currently, various water sports are popular, such as surfing, snorkeling, diving, and underwater photography. In these scenarios, users often use waterproof protective bags to prevent water damage to electronic devices. While waterproof bags offer some protection, significant problems remain in actual water or underwater environments. Due to variations in water pressure and the thickness of the waterproof bag material, touch controls often malfunction, severely impacting the user experience. Typical problems include: Social and communication barriers: Users have difficulty in smoothly initiating or switching video calls and sending multimedia content via touch, resulting in interruptions in the interaction process; Limited shooting capabilities: When shooting underwater, it is impossible to accurately focus, switch shooting modes, or zoom, which affects image quality and creative freedom.

[0003] The aforementioned shortcomings limit users' ability to use electronic devices efficiently and conveniently in complex environments such as aquatic environments. Summary of the Invention

[0004] The purpose of this application is to provide an interaction method, device, and electronic device that enables users to use electronic devices efficiently and conveniently even in complex environments.

[0005] In a first aspect, embodiments of this application provide an interaction method, including: When the camera detects that a target object has entered the optical path area of ​​the auxiliary light source, the first optical flow information of the continuous image frames captured by the camera is obtained. Based on the first optical flow information, perform the corresponding operation on the current display interface.

[0006] Secondly, embodiments of this application provide an interactive device, including: The acquisition module is used to acquire the first optical flow information of continuous image frames captured by the camera when the camera detects that a target object has entered the optical path area of ​​the auxiliary light source; The operation module is used to perform corresponding operations on the current display interface based on the first optical flow information.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0008] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0009] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0010] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0011] In this embodiment, when a target object is detected entering the optical path area of ​​the auxiliary light source by the camera, first optical flow information of continuous image frames captured by the camera is acquired; based on the first optical flow information, corresponding operations are performed on the current display interface. With this solution, the user does not need to rely on screen touch. By transforming the optical path area of ​​the auxiliary light source into a highly sensitive non-touch interactive interface, multiple common operations on the current display interface can be completed through specific gestures or operations of the target object on this optical path area. This achieves accurate and reliable operation of the electronic device, enabling users to use electronic devices efficiently and conveniently even in environments such as water. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating the interaction method provided in some embodiments of this application; Figure 2 These are schematic diagrams showing the underwater mode before and after activation, provided in some embodiments of this application; Figure 3 These are schematic diagrams of the display interface for sliding a target object provided in some embodiments of this application; Figure 4 This is a schematic diagram of the horizontal movement of a target object provided in some embodiments of this application; Figure 5 These are schematic diagrams of continuous image frames provided in some embodiments of this application; Figure 6 These are schematic diagrams showing the focus before and after movement, provided in some embodiments of this application; Figure 7 This is a schematic diagram of double-clicking a target object provided in some embodiments of this application; Figure 8 These are schematic diagrams of continuous image frames provided in some embodiments of this application; Figure 9 These are schematic diagrams of continuous image frames provided in some embodiments of this application; Figure 10 These are schematic diagrams showing the page jump before and after some embodiments of this application; Figure 11 This is a schematic diagram illustrating parameter selection provided in some embodiments of this application; Figure 12 This is a schematic diagram illustrating parameter selection provided in some embodiments of this application; Figure 13 These are schematic diagrams showing the underwater assistance function before and after activation, provided in some embodiments of this application; Figure 14 These are schematic diagrams showing the display before and after clicking on aquatic organisms, provided in some embodiments of this application; Figure 15 These are schematic diagrams of the structure of the interactive device provided in some embodiments of this application; Figure 16 These are structural block diagrams of electronic devices provided in some embodiments of this application; Figure 17 These are structural block diagrams of electronic devices provided in some embodiments of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0014] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0015] The interaction method provided in this application will be described below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0016] like Figure 1 As shown in the figure, this application provides an interaction method, which may specifically include the following steps: Step 101: When the camera detects that the target object has entered the optical path area of ​​the auxiliary light source, the first optical flow information of the continuous image frames captured by the camera is obtained.

[0017] Specifically, when electronic devices are in environments such as water, sandstorms, or heavy rain, and the use of protective measures (e.g., protective bags) makes it inconvenient for users to operate the screen, users need to activate environmental interference modes, such as underwater mode. Figure 2 This is a comparison chart of the current display interface 21 before the underwater mode is turned on and the current display interface 22 after the underwater mode is turned on.

[0018] When the ambient interference mode is enabled, the touch operation function of the screen can be disabled to prevent accidental touches; alternatively, the touch operation function of the screen can be retained so that users can operate the electronic device in multiple ways.

[0019] With the ambient interference mode enabled, the electronic device's camera is activated to capture images, and the auxiliary light source (e.g., flash) is also turned on to provide background illumination for the camera's field of view. After the camera and auxiliary light source are activated, the user can enter the light path area of ​​the auxiliary light source by touching its casing with a target object or by entering it non-contactly. This entry of the target object into the light path area causes a change in the light intensity of the image captured by the camera. The target object can be the user's fingers, palm, fist, etc.

[0020] When the camera detects that a target object has entered the optical path area of ​​the auxiliary light source, the first optical flow information of the continuous image frames captured by the camera is obtained. The first optical flow information includes, but is not limited to, the displacement direction, displacement distance, and pixel value changes of the pixels or feature points in the continuous image frames.

[0021] Users can also operate electronic devices by pressing physical buttons (such as volume buttons, power buttons, etc.).

[0022] Step 102: Based on the first optical flow information, perform the corresponding operation on the current display interface.

[0023] Specifically, the first optical flow information can be used to determine whether the target object is moving, the direction of movement, and other information, thereby determining the corresponding operation and executing the operation corresponding to the first optical flow information on the current display interface.

[0024] In this embodiment, users do not need to rely on screen touch. By transforming the optical path area of ​​the auxiliary light source into a highly sensitive non-touch interactive interface, users can complete a number of common operations on the current display interface through specific gestures or operations of the target object on the optical path area. This enables accurate and reliable operation of electronic devices, allowing users to use electronic devices efficiently and conveniently even in environments such as water.

[0025] In an optional specific embodiment, step 101, when the camera detects that a target object has entered the optical path region of the auxiliary light source, acquires the first optical flow information of consecutive image frames captured by the camera, including: When an object is detected entering the light path area by the camera, the object is identified; If the object is identified as the target object, the first optical flow information of the continuous image frames captured by the camera is obtained, and the distance between the camera and the target object is obtained. Among them, based on the first optical flow information, corresponding operations are performed on the current display interface, including: When the distance is within a preset distance, the corresponding operation is performed on the current display interface based on the first optical flow information.

[0026] Specifically, after the camera and auxiliary light source are turned on, the user can enter the light path area of ​​the auxiliary light source by touching the outer shell of the auxiliary light source or by entering the light path area of ​​the auxiliary light source in a non-contact manner. At this time, since the object enters the light path area, the light intensity of the image captured by the camera changes, and the object needs to be detected and identified to determine whether the object is the target object (e.g., a fist). This can be determined by feature point comparison, etc., without further specific limitations.

[0027] If the object is determined not to be the target object, no response is made to avoid accidental touches. If the object is determined to be the target object, the first optical flow information of the continuous image frames captured by the camera and the distance between the camera and the target object are obtained. It is then determined whether the distance is within a preset distance. If the distance is outside the preset distance, it is determined to be an accidental touch, and no response is made. If the distance is within the preset distance, it is determined to be a non-accidental touch, and the corresponding operation is performed on the current display interface based on the first optical flow information.

[0028] It should be noted that for cases where the distance equals the preset distance, it can be determined as either a mis-touch or a genuine touch, and this can be set as needed. The preset distance is a distance threshold that can be adjusted as required.

[0029] In an optional specific embodiment, the step of performing a corresponding operation on the current display interface based on the first optical flow information when the distance is within a preset distance specifically includes: When the distance is within a preset distance, the third occlusion state sequence of the target object on the optical path area is determined based on the first optical flow information; In the case where the third occlusion state sequence is the occlusion position movement caused by the movement of the target object in the optical path area, a second movement command with focus on the current display interface is determined based on the third occlusion state sequence. In response to the second movement command, update the displayed content of the current screen.

[0030] Specifically, when the distance between the camera and the target object is within a preset distance, the first optical flow information of each image frame can be used to determine whether the target object occludes the optical path area and / or the occlusion position of the optical path area. Thus, the first optical flow information of consecutive image frames can be used to determine the third occlusion state sequence of the target object on the optical path area. The third occlusion state sequence is a sequence composed of multiple consecutive occlusion states, and the occlusion states include: whether the optical path area is occluded and / or the occlusion position of the optical path area.

[0031] Based on the third occlusion state sequence, it can be determined that the camera recognizes a directional gradual change in the intensity of the light source. This further determines the second movement command for the focus on the current display interface. The direction of focus movement indicated by the second movement command is consistent with the direction of change in the intensity of the light source recognized by the camera, so as to facilitate user operation.

[0032] In response to the direction of focus movement indicated by the second movement command, the focus is moved from the current position to the target position, triggering the up-down or left-right swiping operation of the current display interface. The user does not need to rely on screen touch. By transforming the light path area of ​​the auxiliary light source into a highly sensitive non-touch interactive interface, the content displayed on the current display interface can be swiped up, down, left, and right to update the displayed content through the third occlusion state sequence of the target object on the light path area.

[0033] For example: Figure 3 As shown, the target object is a fist 31 formed by clenching a fist, and the arrow indicates the sliding direction of the fist. By sliding the fist 31 up and down, the content displayed on the current display interface 32 will slide up and down accordingly to update the displayed content.

[0034] In the above embodiments, users do not need to rely on screen touch. By transforming the optical path area of ​​the auxiliary light source into a highly sensitive non-touch interactive interface, the user can complete the sliding operation of the current display interface by moving the target object on the optical path area. This enables accurate and reliable operation of electronic devices, allowing users to use electronic devices efficiently and conveniently even in environments such as water.

[0035] In an optional specific embodiment, step 102 performs a corresponding operation on the current display interface based on the first optical flow information, including steps 1021 and 1022: Step 1021: Based on the first optical flow information, determine the first occlusion state sequence of the target object on the optical path region.

[0036] Specifically, the first optical flow information of each image frame can determine whether the target object occludes the optical path region and / or the occlusion position of the optical path region. Thus, the first occlusion state sequence of the target object on the optical path region can be determined by the first optical flow information of consecutive image frames. The first occlusion state sequence is a sequence composed of multiple consecutive occlusion states, and the occlusion state includes: whether the optical path region is occluded and / or the occlusion position of the optical path region.

[0037] Step 1022: Based on the first occlusion state sequence, perform the corresponding operation on the current display interface.

[0038] Specifically, based on the first occlusion state sequence, information such as whether the target object has moved and the direction of movement can be determined, thereby determining the corresponding operation and executing the operation corresponding to the first occlusion state sequence on the current display interface.

[0039] In the above embodiments, users do not need to rely on screen touch. By transforming the optical path area of ​​the auxiliary light source into a highly sensitive non-touch interactive interface, and by using the first occlusion state sequence of the target object on the optical path area, users can complete common operations on the current display interface, thereby achieving accurate and reliable operation of electronic devices. This enables users to use electronic devices efficiently and conveniently even in environments such as water.

[0040] In an optional specific embodiment, step 1022, based on the first occlusion state sequence, performs a corresponding operation on the current display interface, including: In the case where the first occlusion state sequence is the occlusion position movement caused by the movement of the target object in the optical path area, a first movement command with focus on the current display interface is determined based on the first occlusion state sequence. In response to the first move command, the focus is moved from the current interface element of the currently displayed interface to the target interface element. The currently displayed interface includes multiple interface elements with operable attributes, and the current interface element and the target interface element are two of the multiple interface elements with operable attributes.

[0041] Specifically, the content of the currently displayed interface is analyzed to decompose it into multiple independent interface elements, and each interface element is assigned operable attributes. Furthermore, with ambient interference mode enabled, a focus is generated, which is used to move between multiple interface elements based on user input. Example: Figure 2 The box 23; when the focus is on the interface element, activate the operable properties of the interface element; when the focus leaves the interface element, deactivate the operable properties of the interface element.

[0042] Interface elements include, but are not limited to: buttons, text boxes, images, etc.

[0043] If the first occlusion state sequence is a sequence of occlusion positions composed of multiple optical path regions, it can be determined based on the first occlusion state sequence that the camera recognizes a directional gradual change in light intensity. This further determines the first movement command of the focus on the current display interface. The direction of focus movement indicated by the first movement command is consistent with the direction of change in light intensity recognized by the camera, so as to facilitate user operation.

[0044] In response to the direction of focus movement indicated by the first movement command, the focus is moved from the current interface element to the target interface element. The user does not need to rely on screen touch. By transforming the light path area of ​​the auxiliary light source into a highly sensitive non-touch interactive interface, the user can complete the movement operation between multiple interface elements through the first occlusion state sequence of the target object on the light path area.

[0045] For example: Figure 4 As shown, the target object is finger 24, the auxiliary light source is flash 25, finger 24 moves horizontally on the surface of flash 25, and the arrow indicates the direction of movement. Continuous image frames captured by the camera are shown in the image. Figure 5 As shown, the light source intensity can be identified as gradually changing in the direction indicated by the arrow, thus determining that the focus moves in the direction of the arrow. Figure 6 As shown, focus 26 moves from the current interface element 27 to the target interface element 28.

[0046] In an optional specific embodiment, step 1032, based on the first occlusion state sequence, performs a corresponding operation on the current display interface, including: When the first occlusion state sequence is the state switching of the target object performing occlusion and unocclusion (i.e., unoccluded optical path area) on the optical path area, the operation instruction for the current interface element of the current display interface is determined based on the first occlusion state sequence; wherein, the current display interface includes multiple interface elements with operable attributes, and the current interface element is one of the multiple interface elements with operable attributes. In response to an operation command, perform the corresponding operation on the current interface element.

[0047] Specifically, if the first occlusion state sequence is a sequence of state transitions between occlusion and unocclusion in multiple optical path regions, the camera can determine that it has detected a gradual change in light intensity based on the first occlusion state sequence, thereby further determining the operation command for the current interface element. In response to this operation command, the corresponding operation is performed on the current interface element. The user does not need to rely on screen touch; by transforming the optical path region of the auxiliary light source into a highly sensitive non-touch interactive interface, the operation on the current interface element can be completed through the first occlusion state sequence of the target object in that optical path region.

[0048] In an optional specific embodiment, the step of determining the operation instruction for the current interface element of the current display interface based on the first occlusion state sequence specifically includes: Obtain the state at the end of the first occlusion state sequence; When the state at the end of the first occlusion state sequence is occlusion state, the operation instruction for the current interface element is determined to be a selection instruction; When the state at the end of the first occlusion state sequence is the unocclusion state, the operation instruction for the current interface element is determined to be a function trigger instruction; The steps described above for responding to an operation command and performing the corresponding operation on the current interface element specifically include: In response to a function trigger command, jump to the target page corresponding to the current interface element or display multiple parameters corresponding to the current interface element.

[0049] Specifically, if the first occlusion state sequence is a sequence consisting of state transitions between occlusion and de-occlusion in multiple optical path regions, the stable state at the end of the first occlusion state sequence is determined to be either an occlusion state or a de-occlusion state. If the state at the end of the first occlusion state sequence is an occlusion state, then based on the first occlusion state sequence, the operation instruction for the current interface element can be determined to be a selection instruction. In response to this selection instruction, the current interface element is in a selected state. Corresponding to the operation of long-pressing the current interface element, operations such as dragging, removing, sharing, and editing can be performed based on subsequent user input.

[0050] For example: Figure 7As shown, the target object is finger 29, and the auxiliary light source is flash 30. Finger 29 taps the flash twice consecutively, that is, taps the surface of flash 30 and removes it, then taps the surface of flash 30 again without removing it. The continuous image frames captured by the camera are as follows: Figure 8 As shown, it can be identified that the light source only changes in intensity, not in direction. The intensity changes are: strong-weak-strong-weak-weak. Therefore, it can be determined that the focus is still on the current interface element, triggering the function of long-pressing the current interface element for selection.

[0051] If the state at the end of the first occlusion state sequence is the unocclusion state, then based on the first occlusion state sequence, it can be determined that the operation instruction on the current interface element is a function trigger instruction. In response to this function trigger instruction, the user jumps to the target page corresponding to the current interface element. For example, if the current interface element is an application icon, the target page is the main page of the application. Alternatively, in response to the function trigger instruction, multiple parameters corresponding to the current interface element are displayed. For example, if the current interface element is a focus icon, the multiple parameters are multiple focus values; if the current interface icon is a magnification icon, the multiple parameters are multiple magnification values.

[0052] For example: Figure 7 As shown, the target object is finger 29, and the auxiliary light source is flash 30. Finger 29 taps the flash twice consecutively, that is, taps the surface of flash 30 and removes it, then taps the surface of flash 30 again and removes it. The continuous image frames captured by the camera are as follows: Figure 9 As shown, it can be identified that the light source only changes in intensity, not direction. The intensity change is: strong-weak-strong-weak-strong. Therefore, it can be determined that the focus is still on the current interface element, triggering the function of jumping to the target page corresponding to the current interface element. The current display interface 41 before the jump and the target page 42 after the jump are as follows: Figure 10 As shown.

[0053] In the above embodiments, users do not need to rely on screen touch. By transforming the optical path area of ​​the auxiliary light source into a highly sensitive non-touch interactive interface, users can complete operations such as page navigation and displaying optical parameters by switching between the state of the target object blocking and unblocking the optical path area.

[0054] It should be noted that if the first occlusion state sequence is a sequence consisting of a switch between an occlusion state and an unocclusion state, no response will be made to avoid accidental touches.

[0055] In an optional specific embodiment, when displaying multiple parameters corresponding to the current interface element, the method further includes: When a target object is detected entering the optical path area by the camera, the second optical flow information of the continuous image frames captured by the camera is obtained; Based on the second optical flow information, determine the second occlusion state sequence of the target object on the optical path region; Based on the second occlusion state sequence, determine the selection instructions for multiple parameters corresponding to the current interface element; In response to a selection command, the target parameter is determined from among multiple parameters.

[0056] Specifically, when multiple parameters corresponding to the current interface element are displayed on the current display interface, if the camera detects that the target object has entered the optical path area of ​​the auxiliary light source, the second optical flow information of the continuous image frames captured by the camera is obtained. The second optical flow information includes, but is not limited to, the displacement direction, displacement distance, and pixel value change of the pixels or feature points in the continuous image frames.

[0057] The second optical flow information of each image frame can determine whether the target object occludes the optical path region and / or the occlusion position of the optical path region. Thus, the second occlusion state sequence of the target object on the optical path region can be determined by the second optical flow information of consecutive image frames. The second occlusion state sequence is a sequence composed of multiple consecutive occlusion states, and the occlusion states include: whether the optical path region is occluded and / or the occlusion position of the optical path region.

[0058] If the second occlusion state sequence is a sequence of occlusion positions of multiple optical path regions that moves about the occlusion position, it can be determined based on the second occlusion state sequence that the camera recognizes a directional gradual change in the light source intensity. This allows for the further determination of a selection instruction for multiple parameters of the focus. The direction of focus movement indicated by this selection instruction is consistent with the direction of change in the light source intensity recognized by the camera, so as to select the target parameter from multiple parameters.

[0059] Example 1: such as Figure 4 As shown, the target object is finger 24, the auxiliary light source is flash 25, finger 24 moves horizontally on the surface of flash 25, and the arrow indicates the direction of movement. Continuous image frames captured by the camera are shown below. Figure 5 As shown, the light source intensity can be identified as gradually changing in the direction indicated by the arrow. Figure 11 As shown, the current interface element is the focus icon 43 (or the focal length adjuster icon), and the parameter is the focus value. This determines that the focus 44 moves in the direction of the arrow to select the target focus value from multiple focus values. Based on the target focus value, the image displayed on the current display interface is refocused, and the refocused image is then displayed on the current display interface 45.

[0060] Example 2: such as Figure 4 As shown, the target object is finger 24, the auxiliary light source is flash 25, finger 24 moves horizontally on the surface of flash 25, and the arrow indicates the direction of movement. Continuous image frames captured by the camera are shown below. Figure 5As shown, the light source intensity can be identified as gradually changing in the direction indicated by the arrow. Figure 12 As shown, the current interface element is the magnification icon 46, and the parameter is the magnification value. This allows the focus 47 to move in the direction of the arrow to select the target magnification value from multiple magnification values. Based on the target magnification value, the image displayed on the current display interface is processed, and the processed image is then displayed on the current display interface 48.

[0061] In the above embodiments, users do not need to rely on screen touch. By transforming the optical path area of ​​the auxiliary light source into a highly sensitive non-touch interactive interface, the target parameter can be selected from multiple parameters through the second occlusion state sequence of the target object on the optical path area.

[0062] In one optional embodiment, the method further includes: Given that the current environment is aquatic, identify the water depth at the current location, the duration of continuous aquatic exposure, and / or the species of aquatic organisms captured by the camera.

[0063] Specifically, when the electronic device is currently in an aquatic environment, if the user activates the underwater assistance function, the electronic device can identify at least one of the following: the water depth information of the current location of the electronic device, the continuous duration of the electronic device in the aquatic environment, and the category of aquatic (or marine) organisms captured by the camera, in order to improve the user's extremely high shooting needs in the water.

[0064] Users can also take photos using physical buttons, or by double-clicking the area where the target object blocks the light path. The underwater assist function icon 49 before activation and the underwater assist function icon 50 after activation are shown below. Figure 13 As shown.

[0065] The water depth information can be identified using a barometer built into the electronic device. The continuous duration of stay in the aquatic environment can be timed starting from when the environmental disturbance mode is activated. The category of aquatic life can be identified through the aquatic life category displayed in the camera's preview interface, such as coral, fish, sea turtles, starfish, etc.

[0066] Among them, the shooting preview interface refers to the visual area on the screen that displays the camera's field of view in real time and overlays the shooting controls after the electronic device starts the camera function and before the actual image is captured.

[0067] The captured images are saved, and users can retrieve detailed information about them, including water depth at the time the image was taken, the duration of exposure in the aquatic environment, and the specific species of aquatic life captured. Clicking on a species will display its description and a corresponding photograph. For example, such as... Figure 14 As shown, the detailed information in the image includes the specific types of aquatic life captured: various corals (such as: octopus coral, grass coral, leather coral), various fish (such as: clownfish, angelfish, Carlson sea goldfish, shark), etc. Clicking on the Carlson sea goldfish in the fish category will display relevant information about the Carlson sea goldfish and the corresponding photo of the Carlson sea goldfish.

[0068] In summary, this embodiment allows users to operate electronic devices without relying on touchscreens. By transforming the optical path area of ​​the auxiliary light source into a highly sensitive non-touchscreen interface, users can perform multiple common operations on the current display interface through specific gestures or operations on the optical path area of ​​the target object, without requiring additional hardware. This enables accurate and reliable operation of the electronic device, allowing users to use it efficiently and conveniently even in environments such as water. It boasts strong practicality and compatibility. Furthermore, by employing an extremely simplified and intuitive set of gestures (e.g., lateral movement, momentary occlusion, prolonged occlusion), users can quickly master core operations without complex training. The gesture design is highly consistent with the logic of everyday physical movements, significantly reducing the cognitive load during emergency or movement-related situations, ensuring easy use for users of different ages and skill levels. Moreover, this solution is deeply compatible with system accessibility services, requiring no additional application-layer adaptation, and can be directly used in most social, camera, and communication applications.

[0069] The interaction method provided in this application can be executed by an interaction device. This application uses an interaction device executing the interaction method as an example to illustrate the interaction device provided in this application.

[0070] like Figure 15 As shown in the figure, this application embodiment also provides an interactive device 1600, specifically including: The acquisition module 1601 is used to acquire the first optical flow information of continuous image frames captured by the camera when the camera detects that a target object has entered the optical path area of ​​the auxiliary light source. The operation module 1602 is used to perform corresponding operations on the current display interface based on the first optical flow information.

[0071] Optionally, the interactive device 1600 also includes a processing module for activating the camera and auxiliary light source when the environmental interference mode is enabled; Optionally, the operation module 1602 is specifically used for: Based on the first optical flow information, determine the first occlusion state sequence of the target object on the optical path region; Based on the first occlusion state sequence, perform the corresponding operation on the current display interface.

[0072] Optionally, when the operation module 1602 performs the corresponding operation on the current display interface based on the first occlusion state sequence, it is specifically used for: In the case where the first occlusion state sequence is the occlusion position movement caused by the movement of the target object in the optical path area, a first movement command with focus on the current display interface is determined based on the first occlusion state sequence. In response to the first move command, the focus is moved from the current interface element of the currently displayed interface to the target interface element. The currently displayed interface includes multiple interface elements with operable attributes, and the current interface element and the target interface element are two of the multiple interface elements with operable attributes.

[0073] Optionally, when the operation module 1602 performs the corresponding operation on the current display interface based on the first occlusion state sequence, it is specifically used for: When the first occlusion state sequence is a state switch between occlusion and de-occlusion of the optical path area by the target object, the operation instruction for the current interface element of the current display interface is determined based on the first occlusion state sequence; wherein, the current display interface includes multiple interface elements with operable attributes, and the current interface element is one of the multiple interface elements with operable attributes. In response to an operation command, perform the corresponding operation on the current interface element.

[0074] Optionally, when determining the operation instruction for the current interface element of the current display interface based on the first occlusion state sequence, the operation module 1602 is specifically used for: Obtain the state at the end of the first occlusion state sequence; When the state at the end of the first occlusion state sequence is occlusion state, the operation instruction for the current interface element is determined to be a selection instruction; When the state at the end of the first occlusion state sequence is the unocclusion state, the operation instruction for the current interface element is determined to be a function trigger instruction; When the operation module 1602 responds to an operation command and performs a corresponding operation on the current interface element, it is specifically used for: In response to a function trigger command, jump to the target page corresponding to the current interface element or display multiple parameters corresponding to the current interface element.

[0075] Optionally, when displaying multiple parameters corresponding to the current interface element, the processing module is also used to: When a target object is detected entering the optical path area by the camera, the second optical flow information of the continuous image frames captured by the camera is obtained; Based on the second optical flow information, determine the second occlusion state sequence of the target object on the optical path region; Based on the second occlusion state sequence, determine the selection instructions for multiple parameters corresponding to the current interface element; In response to a selection command, the target parameter is determined from among multiple parameters.

[0076] Optionally, module 1601 is used specifically for: When an object is detected entering the light path area by the camera, the object is identified; If the object is identified as the target object, the first optical flow information of the continuous image frames captured by the camera is obtained, and the distance between the camera and the target object is obtained. Among them, based on the first optical flow information, corresponding operations are performed on the current display interface, including: When the distance is within a preset distance, the corresponding operation is performed on the current display interface based on the first optical flow information.

[0077] Optionally, when the acquisition module 1601 performs a corresponding operation on the current display interface based on the first optical flow information, if the distance is within a preset distance, it is specifically used for: When the distance is within a preset distance, the third occlusion state sequence of the target object on the optical path area is determined based on the first optical flow information; In the case where the third occlusion state sequence is the occlusion position movement caused by the movement of the target object in the optical path area, a second movement command with focus on the current display interface is determined based on the third occlusion state sequence. In response to the second movement command, update the displayed content of the current screen.

[0078] The interactive device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the scope of the device.

[0079] The interactive device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0080] The interactive device provided in this application embodiment can achieve... Figures 1 to 14 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0081] Optionally, such as Figure 16 As shown, this application embodiment also provides an electronic device 900, including a processor 901 and a memory 902. The memory 902 stores a program or instructions that can run on the processor 901. When the program or instructions are executed by the processor 901, they implement the various steps of the above-described interaction method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0082] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0083] Figure 17 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0084] The electronic device 1000 includes, but is not limited to, components such as: radio frequency unit 1001, network module 1002, audio output unit 1003, input unit 1004, sensor 1005, display unit 1006, user input unit 1007, interface unit 1008, memory 1009, and processor 1010.

[0085] Those skilled in the art will understand that the electronic device 1000 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1010 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 17 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0086] The processor 1010 is used to acquire first optical flow information of continuous image frames captured by the camera when the camera detects that a target object has entered the optical path area of ​​the auxiliary light source. Based on the first optical flow information, perform the corresponding operation on the current display interface.

[0087] Optionally, the processor 1010 is also used to turn on the camera and auxiliary light source when the ambient interference mode is enabled; Optionally, when the processor 1010 performs a corresponding operation on the current display interface based on the first optical flow information, it is specifically used for: Based on the first optical flow information, determine the first occlusion state sequence of the target object on the optical path region; Based on the first occlusion state sequence, perform the corresponding operation on the current display interface.

[0088] Optionally, when the processor 1010 performs the corresponding operation on the current display interface based on the first occlusion state sequence, it is specifically used for: In the case where the first occlusion state sequence is the occlusion position movement caused by the movement of the target object in the optical path area, a first movement command with focus on the current display interface is determined based on the first occlusion state sequence. In response to the first move command, the focus is moved from the current interface element of the currently displayed interface to the target interface element. The currently displayed interface includes multiple interface elements with operable attributes, and the current interface element and the target interface element are two of the multiple interface elements with operable attributes.

[0089] Optionally, when the processor 1010 performs the corresponding operation on the current display interface based on the first occlusion state sequence, it is specifically used for: When the first occlusion state sequence is a state switch between occlusion and de-occlusion of the optical path area by the target object, the operation instruction for the current interface element of the current display interface is determined based on the first occlusion state sequence; wherein, the current display interface includes multiple interface elements with operable attributes, and the current interface element is one of the multiple interface elements with operable attributes. In response to an operation command, perform the corresponding operation on the current interface element.

[0090] Optionally, when the processor 1010 determines the operation instruction for the current interface element of the current display interface based on the first occlusion state sequence, it specifically performs the following: Obtain the state at the end of the first occlusion state sequence; When the state at the end of the first occlusion state sequence is occlusion state, the operation instruction for the current interface element is determined to be a selection instruction; When the state at the end of the first occlusion state sequence is the unocclusion state, the operation instruction for the current interface element is determined to be a function trigger instruction; When processor 1010 responds to an operation instruction and performs a corresponding operation on the current interface element, it is specifically used for: In response to a function trigger command, jump to the target page corresponding to the current interface element or display multiple parameters corresponding to the current interface element.

[0091] Optionally, when displaying multiple parameters corresponding to the current interface element, the processor 1010 is also configured to: When a target object is detected entering the optical path area by the camera, the second optical flow information of the continuous image frames captured by the camera is obtained; Based on the second optical flow information, determine the second occlusion state sequence of the target object on the optical path region; Based on the second occlusion state sequence, determine the selection instructions for multiple parameters corresponding to the current interface element; In response to a selection command, the target parameter is determined from among multiple parameters.

[0092] Optionally, when the processor 1010 detects that a target object has entered the optical path region of the auxiliary light source, and acquires the first optical flow information of consecutive image frames captured by the camera, it is specifically used for: When an object is detected entering the light path area by the camera, the object is identified; If the object is identified as the target object, the first optical flow information of the continuous image frames captured by the camera is obtained, and the distance between the camera and the target object is obtained. Among them, based on the first optical flow information, corresponding operations are performed on the current display interface, including: When the distance is within a preset distance, the corresponding operation is performed on the current display interface based on the first optical flow information.

[0093] Optionally, when the processor 1010 performs a corresponding operation on the current display interface based on the first optical flow information, when the distance is within a preset distance, it is specifically used for: When the distance is within a preset distance, the third occlusion state sequence of the target object on the optical path area is determined based on the first optical flow information; In the case where the third occlusion state sequence is the occlusion position movement caused by the movement of the target object in the optical path area, a second movement command with focus on the current display interface is determined based on the third occlusion state sequence. In response to the second movement command, update the displayed content of the current screen.

[0094] It should be understood that, in this embodiment, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042. The GPU 10041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1006 may include a display panel 10061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 1007 includes at least one of a touch panel 10071 and other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, joysticks, etc., which will not be described in detail here.

[0095] The memory 1009 can be used to store software programs and various data. The memory 1009 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1009 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1009 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0096] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor 1010.

[0097] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described interaction method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0098] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0099] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described interaction method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0100] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0101] This application provides a computer program product that is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described interactive method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0102] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0104] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An interaction method, characterized in that, include: When a target object is detected by the camera entering the optical path area of ​​the auxiliary light source, the first optical flow information of the continuous image frames captured by the camera is obtained; Based on the first optical flow information, perform the corresponding operation on the current display interface.

2. The method according to claim 1, characterized in that, The step of performing corresponding operations on the current display interface based on the first optical flow information includes: Based on the first optical flow information, a first occlusion state sequence of the target object on the optical path region is determined; Based on the first occlusion state sequence, perform the corresponding operation on the current display interface.

3. The method according to claim 2, characterized in that, The step of performing corresponding operations on the current display interface based on the first occlusion state sequence includes: When the first occlusion state sequence is the occlusion position movement caused by the movement of the target object in the optical path area, a first movement command of the focus on the current display interface is determined based on the first occlusion state sequence. In response to the first movement command, the focus is moved from the current interface element of the currently displayed interface to the target interface element of the currently displayed interface.

4. The method according to claim 2, characterized in that, The step of performing corresponding operations on the current display interface based on the first occlusion state sequence includes: When the first occlusion state sequence is a state switch where the target object performs occlusion and de-occlusion on the optical path area, an operation instruction for the current interface element of the current display interface is determined based on the first occlusion state sequence. In response to the operation instruction, the corresponding operation is performed on the current interface element.

5. The method according to claim 4, characterized in that, The step of determining the operation instruction for the current interface element of the current display interface based on the first occlusion state sequence includes: Obtain the state at the termination of the first occlusion state sequence; When the state at the end of the first occlusion state sequence is an occlusion state, the operation instruction for the current interface element is determined to be a selection instruction; When the state at the end of the first occlusion state sequence is the unocclusion state, the operation instruction on the current interface element is determined to be a function trigger instruction; The operation performed on the current interface element in response to the operation instruction includes: In response to the function trigger command, the user is redirected to the target page corresponding to the current interface element or multiple parameters corresponding to the current interface element are displayed.

6. The method according to claim 5, characterized in that, When displaying multiple parameters corresponding to the current interface element, the method further includes: When the camera detects that the target object has entered the optical path region, the second optical flow information of the continuous image frames captured by the camera is obtained; Based on the second optical flow information, a second occlusion state sequence of the target object on the optical path region is determined; Based on the second occlusion state sequence, determine the selection instructions for multiple parameters corresponding to the current interface element; In response to the selection instruction, a target parameter among the plurality of parameters is determined.

7. The method according to claim 1, characterized in that, The step of acquiring the first optical flow information of consecutive image frames captured by the camera when the camera detects that a target object has entered the optical path area of ​​the auxiliary light source includes: When the camera detects that an object has entered the optical path area, the object is identified. If the object is identified as the target object, the first optical flow information of the continuous image frames captured by the camera is obtained, and the distance between the camera and the target object is obtained. The step of performing corresponding operations on the current display interface based on the first optical flow information includes: If the distance is within a preset distance, the corresponding operation is performed on the current display interface based on the first optical flow information.

8. The method according to claim 7, characterized in that, When the distance is within a preset distance, based on the first optical flow information, the corresponding operation is performed on the current display interface, including: When the distance is within a preset distance, a third occlusion state sequence of the target object on the optical path region is determined based on the first optical flow information; When the third occlusion state sequence is a movement of the occlusion position caused by the movement of the target object in the optical path area, a second movement command for the focus on the current display interface is determined based on the third occlusion state sequence. In response to the second movement command, the display content of the currently displayed interface is updated.

9. An interactive device, characterized in that, include: The acquisition module is used to acquire the first optical flow information of the continuous image frames captured by the camera when the camera detects that a target object has entered the optical path area of ​​the auxiliary light source; The operation module is used to perform corresponding operations on the current display interface based on the first optical flow information.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the interactive method as described in any one of claims 1-8.

Images