Interaction conflict resolution method and electronic device

Abstract

An interaction conflict resolution method and an electronic device, relating to the technical field of terminals. The method comprises: displaying a first interface, wherein a first video is being played in the first interface; in response to a long press operation on the first video, controlling the playback of the first video; displaying a second interface, wherein in the second interface, the playback of a second video is paused; and in response to a long press and drag operation on the second video, displaying quick access entries of a plurality of first services while dragging and moving a thumbnail of the second video. In this way, conflicts between video playback control and magic portal responses can be avoided, thereby improving the accuracy of responses.

Description

An interaction conflict resolution method and electronic device

[0001] The present application claims priority to the Chinese patent application No. 202411255311.2, filed on September 6, 2024, and entitled "An interaction conflict resolution method and electronic device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of terminals, and in particular to an interaction conflict resolution method and electronic device. BACKGROUND

[0003] Electronic devices such as mobile phones and tablets can provide an arbitrary door function. The arbitrary door is a human-computer interaction mode based on intent recognition, which can enable services to achieve one-step direct access across applications. A typical response process of the arbitrary door is as follows: after detecting the user's long press and drag operation, the electronic device can recognize the user's intent, and based on the result of intent recognition, provide multiple candidate services on the edge of the display screen. In response to the user's selection of the target service, the target service can be directly accessed.

[0004] However, the long press operation in the above long press and drag operation can conflict with the triggering operation of other functions of the electronic device. In these conflict scenarios, how to respond is a problem to be solved. SUMMARY

[0005] The present application provides an interaction conflict resolution method and electronic device, which can avoid the problem of conflict between the arbitrary door and the triggering operation of other functions, and achieve accurate response of the arbitrary door and other functions.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0007] In a first aspect, the present application provides an interaction conflict resolution method. The method can be executed by an electronic device, or by a component of the electronic device, such as a processor, a chip, or a chip system of the electronic device, or by a logic module or software that can realize all or part of the electronic device. Hereinafter, the method is taken as an example to be executed by the electronic device. Specifically, the method comprises:

[0008] The first interface is displayed, and the first video is playing in the first interface. The first interface can be interface 702, application interface 9021, etc. below. In response to the long press operation on the first video, the playing of the first video is controlled, such as fast forwarding, speed changing, etc. The second interface is displayed, and the second video is paused in the second interface. The second interface can be application interface 9022 below. In this way, in the case that the video itself can control the playing of the video in response to the long press operation, the playing control of the video can be performed without responding to any door, so as to avoid conflicts.

[0009] In response to the long press and drag operation on the second video, the thumbnail of the second video is moved along with the dragging, such as the thumbnail of the puppy in interface 904, and the shortcut entry of the plurality of first services is displayed. For example, the plurality of first services are services matched with the second video or a frame image in the second video, such as service 111 and service 222 in interface 904. In this way, in the case that the playing of the video does not need to be controlled, there is no conflict, and at this time, any door can be responded to, so as to provide the user with a shortcut service matched with the intention, so that the user can directly reach the corresponding service in one step. For example, in response to the user releasing the second video thumbnail after dragging the second video thumbnail to the shortcut entry of the target service, the interface of the target service can be displayed.

[0010] In summary, by using the present application, in the conflict scenario of playing a video, the normal response of the control function of the video can be ensured, and conflicts can be avoided. In the non-conflict scenario of pausing the playing of the video, the drag response of any door can be implemented, the user can use the required service conveniently, and the efficiency of human-computer interaction is improved.

[0011] In a possible design manner of the first aspect, the playing of the first video is controlled, including: fast forwarding the first video.

[0012] In a possible design manner of the first aspect, the first interface and the second interface are not application interfaces of a gallery application. For example, the first interface and the second interface can be interfaces of a video player, a browser interface, a video panel interface of a social chat application, etc.

[0013] Generally, when a video is played in a gallery application, the gallery application does not have a special response to the long press operation of the user. That is, when a video is played in a gallery application, there is generally no conflict with any door, and accordingly, in the gallery application, any door can be directly responded to without being based on whether a video is played.

[0014] In a possible design of the first aspect, the first video and the second video are the same video, as shown in the scene from the interface 702 to the interface 802 to the interface 803 in FIG. 8. After the playing of the first video is controlled in response to the long-press operation on the first video, the second interface is displayed, including: in response to the pause playing operation on the first video, the second interface is displayed.

[0015] That is, for the same video, different responses can be performed after the long-press operation in the playing state and the pause state, so that targeted responses in different states are realized.

[0016] In a possible design of the first aspect, the first interface is displayed, and the second interface is displayed, including: the first interface and the second interface are simultaneously displayed in the foreground, as shown in the scene from the interface 902 to the interface 904 in FIG. 9.

[0017] That is, for the two videos that are simultaneously displayed, different responses can be performed after the long-press operation due to different states, so that targeted responses in different states are realized.

[0018] In a possible design of the first aspect, the playing of the first video is controlled in response to the long-press operation on the first video, including: in response to the long-press operation on the first video, it is detected that the pressed video is a video being played, and the playing of the first video is controlled.

[0019] In a possible design of the first aspect, the shortcut entry of the plurality of first candidate services is displayed in response to the long-press and drag operation on the second video, and the thumbnail of the second video is moved along with the dragging, including: in response to the long-press and drag operation on the second video, it is detected that the pressed video is not a video being played, and the shortcut entry of the plurality of first services is displayed along with the moving of the thumbnail of the second video.

[0020] That is, after the long-press operation on the video is detected, it can be determined how to respond by detecting whether the pressed video is a video being played, so that accurate responses for different states can be realized.

[0021] In a possible design of the first aspect, the first interface further includes first content, and the first content is not a video. The first content can be a picture, text, or the like. For example, the first content is the text area 7022 in the interface 702. After the first interface is displayed, the method further includes: in response to the long-press and drag operation on the first content, it is detected that the pressed video is not a video being played, and the thumbnail of the first content is moved along with the dragging, and the shortcut entry of the plurality of second services is displayed.

[0022] That is, the long press operation on the non-video can also trigger detection of whether the pressing is on a video being played, and can get a result that the pressing is not on a video being played, so that the drag of the arbitrary gate can be responded to.

[0023] It should be noted that the response to the long press operation on the non-video content in the second interface is the same, which will not be described here.

[0024] In a possible design of the first aspect, the pressing is detected to be on a video being played in a case where at least one of the following conditions (hereinafter referred to as conflict condition 1) is met: the video is being played (1); the pressing is on a video playing window (2); and the application on which the pressing is performed and the application on which the video is being played are the same application (3).

[0025] In combination with (3), in a scenario where multiple applications are running in the foreground, the conflict can be more accurately identified. For example, in the following case 4, the video on which the pressing is performed and the video being played are not the same, so it can be detected that the pressing is not on a video being played, that is, it is identified that there is no conflict. Therefore, the drag of the arbitrary gate can be responded to to the greatest extent, and the human-computer interaction efficiency is improved.

[0026] In a possible design of the first aspect, after displaying the shortcut entry of the plurality of first services, the method further includes: displaying a third interface, the third interface including second content. Receiving a long press operation of the user on the second content. In a case where it is detected that the conflict condition is met, performing a long press response of the second content or a drag response. In a case where it is detected that the conflict condition is not met, in response to the drag operation on the second content, moving a thumbnail of the second content along with the drag, and displaying a shortcut entry of a plurality of third services.

[0027] The conflict condition includes one or more of the following conflict conditions 2 to 5.

[0028] In the following scenario two, the third interface can be interface 1001 in FIG. 10, interface 1102 in FIG. 11, and the like, and correspondingly, the second content can be the content in these interfaces, and the long press response can be a recording / video recording related control function.

[0029] In the following scenario three, the third interface can be interface 1201 in FIG. 12, the second content can be the content in interface 1201, and the drag response can be deletion, order adjustment, and the like.

[0030] In the following scenario four, the third interface can be interface 1301 in FIG. 13, interface 1401 in FIG. 14, and the like, and correspondingly, the second content can be the content in these interfaces, and the long press response can be a game operation, such as fast forward, continuous shooting, and the like.

[0031] In the following scenario five, the third interface can be the interface 1501 in FIG. 15, the interface 1601 in FIG. 16, etc., and correspondingly, the second content can be the content in these interfaces, and the long press response can be adjusting the order, etc.

[0032] That is, after detecting the long press operation, it can be detected whether there is a conflict through a conflict condition, and different responses can be made for the case where there is a conflict and the case where there is no conflict, so that the conflict can be avoided, and the human-computer interaction efficiency can be improved in the case where there is no conflict.

[0033] In a possible design of the first aspect, the conflict condition includes at least one of the following:

[0034] The target application is an application that is calling the microphone to collect sound and / or the camera to collect images, the target application is the application corresponding to the third interface, that is, the application that calls the microphone to collect sound (that is, a recording scene) and / or calls the camera to collect images (that is, a video recording scene) performs the long press operation, corresponding to the conflict condition 2 in the following.

[0035] The second content supports the drag response, that is, the long press operation is performed on the control that supports the drag response by itself, corresponding to the conflict condition 3 in the following.

[0036] The third interface includes a game screen, and the second content is the content in the game screen, that is, the long press operation is performed on the game screen, corresponding to the conflict condition 4 in the following.

[0037] The second content is a preset constituent element in the third interface, such as a title bar, a selection button, etc., that is, the long press operation is performed on the preset constituent element in the interface, corresponding to the conflict condition 5 in the following.

[0038] In a possible design of the first aspect, in response to the drag operation on the second content, a thumbnail of the second content is moved along with the drag, and a shortcut entry of a plurality of third services is displayed, including: in response to the drag operation on the second content, a thumbnail of the second content is moved along with the drag, and a shortcut entry of a plurality of third services is displayed, after the long press operation on the second content reaches a first time length. Before the long press operation on the second content reaches the first time length, the method further includes: displaying a long press menu bar of a target application after the long press operation on the second content reaches a second time length, the target application being an application corresponding to the third interface.

[0039] In the second aspect, the present application also provides an electronic device, which includes a display screen, a memory and one or more processors. The display screen, the memory and the processor are coupled. The memory is used to store computer program code, the computer program code includes computer instructions, when the computer instructions are executed by the processor, the electronic device executes the method in the first aspect and any possible design thereof.

[0040] In a third aspect, the present application provides a chip system, which is applied to an electronic device including a display screen and a memory; the chip system includes one or more interface circuits and one or more processors; the interface circuits and the processors are interconnected through lines; the interface circuits are configured to receive signals from the memory of the electronic device and send signals to the processors, the signals including computer instructions stored in the memory; when the processors execute the computer instructions, the electronic device performs the method according to the first aspect and any possible design of the first aspect.

[0041] In a fourth aspect, the present application provides a computer storage medium, which includes computer instructions, when the computer instructions are executed on an electronic device, the electronic device performs the method according to the first aspect and any possible design of the first aspect.

[0042] In a fifth aspect, the present application provides a computer program product, when the computer program product is executed on a computer, the computer performs the method according to the first aspect and any possible design of the first aspect.

[0043] It can be understood that the electronic device provided in the second aspect, the chip system provided in the third aspect, the computer storage medium provided in the fourth aspect, and the computer program product provided in the fifth aspect can achieve the beneficial effects of the first aspect and any possible design of the first aspect, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0044] FIG. 1 is a schematic diagram of a response object category of an arbitrary gate according to an embodiment of the present application;

[0045] FIG. 2 is an interaction diagram of a taxi-booking scheme according to an embodiment of the present application;

[0046] FIG. 3 is an interaction diagram of another taxi-booking scheme according to an embodiment of the present application;

[0047] FIG. 4 is a schematic diagram of a conflict situation according to an embodiment of the present application;

[0048] FIG. 5 is a hardware structure diagram of an electronic device according to an embodiment of the present application;

[0049] FIG. 6 is a framework structure diagram of a service recommendation according to an embodiment of the present application;

[0050] FIG. 7 is a response diagram of a video scene according to an embodiment of the present application;

[0051] FIG. 8 is a response diagram of another video scene according to an embodiment of the present application;

[0052] FIG. 9 is a response diagram of still another video scene according to an embodiment of the present application;

[0053] Fig. 10 is a response diagram of a recording scene according to an embodiment of the present application;

[0054] Fig. 11 is a response diagram of another recording scene according to an embodiment of the present application;

[0055] Fig. 12 is a response diagram of a control supporting drag response according to an embodiment of the present application;

[0056] Fig. 13 is a response diagram of a game scene according to an embodiment of the present application;

[0057] Fig. 14 is a response diagram of another game scene according to an embodiment of the present application;

[0058] Fig. 15 is a response diagram of a title bar according to an embodiment of the present application;

[0059] Fig. 16 is a response diagram of a selection button according to an embodiment of the present application;

[0060] Fig. 17 is a flowchart of a conflict resolution method according to an embodiment of the present application;

[0061] Fig. 18 is a classification diagram of a conflict response according to an embodiment of the present application;

[0062] Fig. 19 is a response diagram of a one-time long press according to an embodiment of the present application. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In the description of the present application, unless otherwise specified, " / " represents an "or" relationship between the objects associated before and after, for example, A / B can represent A or B; in the present application, "and / or" is only a description of the associated relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone, where A and B can be singular or plural. In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. "At least one of the following" or similar expressions means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b, or c, can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple. In addition, in order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, "first", "second", and the like are used to distinguish the same items or similar items with basically the same function and role. Those skilled in the art can understand that "first", "second", and the like do not limit the quantity and execution order, and "first", "second", and the like do not necessarily mean different.

[0064] At the same time, in the embodiments of the present application, the words "exemplary" or "for example" are used to represent an example, illustration or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. On the contrary, the words "exemplary" or "for example" are used to present the relevant concept in a specific way, for easy understanding.

[0065] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0066] Mobile phones, tablets and other electronic devices can provide the function of any door, which is a human-computer interaction mode based on intent recognition, and can make services achieve one-step direct access across applications. Typically, a user can make services achieve one-step direct access across applications by one press (long press) and one drag (drag, drag), thereby simplifying the service access process.

[0067] Referring to FIG. 1, after detecting the long press of the user on the picture, the text, the electronic device can further respond to the drag operation to trigger the arbitrary gate. After detecting the long press of the user on the blank area, the operation button, the electronic device does not further respond to the drag operation, and the arbitrary gate is not triggered.

[0068] At this point, it should be noted that the object type that the arbitrary gate can respond to is not limited to the example shown in FIG. 1. In practice, the object type that can be responded to can also be a video, an audio, a document (such as a word document, a PPT document, etc.), a folder, a desktop, and even some operation buttons.

[0069] Among them, the arbitrary gate can be used in the use scenarios of the services such as taking a taxi, sharing, and searching.

[0070] Next, taking the use scenario of taking a taxi in a mobile phone as an example, the conventional taxi taking scheme and the taxi taking scheme through the arbitrary gate are compared.

[0071] In the conventional taxi taking scheme, the user can first determine the destination, such as copying the destination address, and then start the taxi taking application, and input the address in the destination of the taxi taking service in the taxi taking application, such as pasting the destination address, to take a taxi.

[0072] Referring to FIG. 2, after receiving the information including the address information, the mobile phone can display an interface 201, and the short message 2011 in the interface 201 includes the address 2012. In the interface 201, the mobile phone can copy the address 2012.

[0073] For example, the mobile phone can first complete the copying of the short message 2011. For example, the mobile phone can provide a copy control in response to the long press operation of the user on the short message 2011, and in response to the triggering operation of the user on the copy control, the mobile phone can copy the content of the short message 2011 to the clipboard. Subsequently, the mobile phone can paste the short message 2011 into the input box 2013 of the interface 201. For example, in response to the long press operation of the user on the input box 2013, the mobile phone can provide a paste control, and in response to the triggering operation of the user on the paste control, the mobile phone can paste the short message 2011 in the clipboard into the input box 2013. Then, the mobile phone can copy the address 2012 from the input box 2013. For example, in response to the triggering operation of the user on the short message 2011 in the input box 2013, the mobile phone can provide a selection control, and in response to the triggering operation of the user on the selection control, the operation of selecting the address 2012 in the short message 2011 in the input box 2013 and then copying, the mobile phone can copy the address 2012.

[0074] Continuing to refer to FIG. 2, after copying the address 2012, in response to a swiping operation from the bottom of the interface 201, the mobile phone can display an interface 202, which is the desktop of the mobile phone. The interface 202 includes an application icon 2021 of the taxi-hailing application. In response to a triggering operation of the application icon 2021 by the user, the mobile phone can display an interface 203, which is an application interface of the taxi-hailing application, i.e., switching to running the taxi-hailing application in the foreground.

[0075] Continuing to refer to FIG. 2, in the interface 203, the mobile phone can receive the address 2012 input by the user. After inputting the address 2012, the mobile phone can display an interface 204.

[0076] For example, in response to a long-press operation of a destination input box 2031 in the interface 203 by the user, the mobile phone can provide a paste control. In response to a triggering operation of the paste control by the user, the mobile phone can paste the address 2012 into the destination input box 2031, thereby achieving the input of the destination.

[0077] In the above conventional taxi-hailing scheme, the actions performed by the user include at least the following: determining the destination, starting the taxi-hailing application, and inputting the destination. Each of the actions further includes one or more sub-actions. For example, determining the destination includes copying and pasting information, and further copying the address from the pasted information. For another example, starting the taxi-hailing application includes returning from the information application to the desktop, and then entering the taxi-hailing application from the desktop.

[0078] In the scheme of taking a taxi through any door, in response to a long-press and drag operation of information including an address by the user, the mobile phone can recognize the intention of taking a taxi and provide an entry of the taxi-hailing service. In response to the user releasing the information including the address at a position of the entry of the taxi-hailing service, the mobile phone can display an application interface of the taxi-hailing application and display a route from the current location to the address included in the information in the application interface.

[0079] Referring to FIG. 3, after receiving information including address information, the mobile phone can display an interface 201, in which a message 2011 includes an address 2012. In response to a long-press and drag operation of the message 2011 by the user, the mobile phone can display an interface 301, which provides shortcut entries corresponding to intentions recognized by the mobile phone, such as a shortcut entry 3011 of the taxi-hailing service, a shortcut entry 3012 of creating a schedule, a shortcut entry 3013 of address search, a shortcut entry 3014 of sharing with friends, a shortcut entry 3015 of creating a note, etc. In response to a releasing operation of the message 1011 by the user after dragging the message 1011 to the shortcut entry 3011 of the taxi-hailing service, the mobile phone can display an interface 302. The interface 302 includes an application interface 3021 of the taxi-hailing application in the form of a floating window, and the application interface 3021 includes a route, a vehicle type, a price, etc. of taking a taxi to the address 2012.

[0080] The action performed by the user in the above solution of taking a car through any door includes long pressing and dragging to the corresponding shortcut (such as the shortcut 3011 of the car service mentioned above) and then releasing the hand.

[0081] Obviously, compared with the conventional car taking solution, the solution of taking a car through any door can greatly simplify the user operation, so that the intention can be realized more quickly.

[0082] At this point, it should be noted that any door is only a name for the visualization of the response process based on the interface (such as the effect of the "door" being pushed open in the interface 302 mentioned above). In practice, the function of achieving one-step direct service based on intention recognition can also be other names, such as "direct service" and "intention recommendation", which are not limited in the present application. In this paper, the naming of any door is mainly taken as an example for explanation, and it should be understood that if it is other naming, the implementation principle of the solution is the same.

[0083] However, the long pressing and dragging operation for triggering the any door may conflict with the triggering operation of some other functions of the electronic device.

[0084] Still taking a mobile phone as an example, referring to FIG. 4, the mobile phone can display an interface 401, which includes a video playing window 4011 in which a video is being played. Generally, in response to the long pressing operation of the user on the video playing window 4011 in the interface 401, the mobile phone can fast forward the video. At the same time, as known from the foregoing description, long pressing and dragging can also trigger the any door. Therefore, for the long pressing operation of the user on the video playing window 4011 in the interface 401, the mobile phone cannot determine whether to provide the fast forward function or further respond to the dragging operation of the any door, and therefore a conflict exists.

[0085] In similar conflict scenarios such as that shown in FIG. 4, how the electronic device responds after receiving the long pressing operation of the user has become a problem to be solved urgently.

[0086] It should be noted that the long pressing and dragging operation can be long pressing and then dragging, or dragging while pressing, which is not limited in the present application.

[0087] At this point, it should be noted that in the above examples, the triggering operation of the any door is taken as an example of long pressing and dragging operation, which is not limited in practice. For example, the triggering operation of the any door can also be the operation of drawing a circle with a finger joint. In this paper, the long pressing and dragging operation is mainly taken as an example for explanation, and it should be understood that if it is other triggering operation, the implementation principle of the solution is the same, and only the conflicting function will be different. For example, if the triggering operation of the any door is the operation of drawing a circle with a finger joint, the conflicting function can be other functions triggered by drawing a circle with a finger joint.

[0088] Based on the above conflict problem, the embodiment of the present application provides an interaction conflict resolution method, which can identify whether there is a conflict, and in the case of conflict, the electronic device can normally provide other functions. In addition, in the case of no conflict, the electronic device can provide the function of the arbitrary door. In this way, the conflict between other functions and the triggering operation of the arbitrary door can be avoided, the accurate response of other functions and the arbitrary door is realized, and different needs of users in different scenarios are met.

[0089] Exemplarily, the electronic device can be a mobile phone, a tablet, a desktop, a laptop, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, and an electronic device providing a function of an arbitrary door, such as a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) \ virtual reality (VR) device, and the like. The embodiment of the present application does not specially limit the specific form of the electronic device. Hereinafter, the electronic device is taken as an example of a mobile phone to illustrate the solution of the present application.

[0090] Referring to FIG. 5, the mobile phone can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, a loudspeaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor 180, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, and the like. The sensor 180 can further include a pressure sensor 180A, a touch sensor 180K.

[0091] It can be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the mobile phone. In other embodiments of the present application, the mobile phone can include more or fewer components than the illustration, or combine certain components, or split certain components, or different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0092] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0093] In a specific implementation, the mobile phone can implement the conflict resolution method through the processor 110 to realize accurate response of other functions and any door, and meet different needs of users in different scenarios.

[0094] The wireless communication function of the mobile phone can be realized through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc.

[0095] The mobile phone realizes the display function through the GPU, the display screen 194, and the application processor, etc. The GPU is a microprocessor for image processing, connected with the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs, which execute program instructions to generate or change display information.

[0096] The display screen 194 is used to display images, videos, etc. In a specific implementation, the mobile phone can display the response effect of any door through the display screen 194, such as providing a shortcut entry, etc.

[0097] The mobile phone can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor, etc.

[0098] The pressure sensor 180A is used to sense the pressure signal, and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be arranged on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. When a touch operation acts on the display screen 194, the mobile phone detects the touch operation intensity according to the pressure sensor 180A.

[0099] Touch sensor 180K, also referred to as a "touch panel". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 together form a touch screen, also referred to as a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the touch event type. In other embodiments, the touch sensor 180K can also be disposed on the surface of the mobile phone, which is different from the position where the display screen 194 is located.

[0100] In a specific implementation, the mobile phone can detect user operations through the above-mentioned pressure sensor 180A and touch sensor 180K, such as detecting long press, drag, and other operations.

[0101] The motor 191 can be used for incoming call vibration prompt, and also can be used for touch vibration feedback. The indicator 192 can be an indicator light, which can be used to indicate the charging state, the power change, and also can be used to indicate messages, missed calls, notifications, and the like. The SIM card interface 195 is used to connect the SIM card. The SIM card can be inserted into or pulled out of the SIM card interface 195 to realize contact and separation with the mobile phone.

[0102] The software system of the mobile phone can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiments of the present application do not make specific limitations on this.

[0103] Embodiments of the present application provide a framework for recommending services in any door, which can solve the above problems.

[0104] Referring to FIG. 6, FIG. 6 exemplarily shows the structure of the framework for recommending services. The framework can include two parts, one part is located on the electronic device side, and the other part is located on the cloud device side.

[0105] As shown in FIG. 6, the electronic device side can include a display entry, a software development kit (SDK), a multi-modal recommendation service, and a service management framework. The multi-modal recommendation service can also be referred to as a multi-modal service recommendation engine, which can include the following modules: a service subscription module, a multi-modal content understanding module, an intent recognition module, a service recall module, a service ranking module, a service notification module, an intent self-learning module, and the like.

[0106] The display entry in the electronic device side can be located in the application layer, and the remaining modules can be located in the framework layer or the service layer, which is not limited here. The display entry can refer to the entry of each service that can provide a service recommendation function, such as copying the entry of a recommended service, dragging the entry of a content recommendation service in the “intelligent flow” function, and receiving the input content of the user. The relationship between the display entry and the interaction mode mentioned above is not limited to the interaction mode mentioned above.

[0107] In some embodiments, the display entry can perform conflict decision, that is, identify whether there is a conflict, and further respond to the drag of any gate in the case that there is no conflict.

[0108] The SDK is used to provide an interface for the display entry to use the multi-modal recommendation service. The framework shown in FIG. 6 defines a unified multi-modal content input interface (i.e., the display entry), and the display module display entry can obtain the input content from the clipboard and the drag event, and then transmit the input content to the SDK, thereby triggering the service recommendation performed by the subsequent multi-modal recommendation service. In this way, the multi-modal recommendation service does not need to obtain the content from the clipboard and the drag event, thereby improving the processing efficiency of information.

[0109] The multi-modal recommendation service is used to recommend services for each business in the application layer. Specifically, the service subscription module is used to subscribe to services and / or intent categories supported by each business, the multi-modal content understanding module is used to identify entities from the content received by the application layer business, the intent identification module is used to find the corresponding intent according to the identified entity, the service recall module is used to find the service corresponding to the intent, the service sorting module is used to sort each service, the service notification module is used to deliver the sorted service list to each business in the application layer, and the intent self-learning module is used to learn new intent categories.

[0110] As shown in FIG. 6, the cloud device side can include a decision cloud platform and an intelligent service platform (ISP).

[0111] The decision cloud platform can be a physical server, a cloud server, a server cluster, etc. The decision cloud platform can be uniformly provided by the device manufacturer of the electronic device.

[0112] The decision cloud platform is used to define the following items:

[0113] 1. Global intent system

[0114] The global intent system includes information of multiple intent categories. The intent refers to the purpose that the user wants to achieve. The intent can be classified into multiple categories, for example, can include but is not limited to opening password, navigation, map search, taking a taxi, shopping, ordering takeout, etc. One intent category can also be understood as one less specific intent.

[0115] Each type of intent can include one or more of the following information: intent identification, intent level, intent tag, intent slot information (slotInfoList). Among them, the intent identification indicates the intent category, which can include one or more of the following: intent identity (ID), intent code. The intent level represents the confidence of the intent.

[0116] The intent slot information can include one or more of the following: slot name, data type filled in the slot, Chinese name of data filled in the slot, and indication information of whether the slot is necessary. The slot can be regarded as a parameter, and the data type can include string, number, etc.

[0117] What the decision cloud platform does is to define the information included in multiple types of intents, such as intent identification and intent slot information. Different types of intents include different information. In other words, the decision cloud platform standardizes the expression form of the intent, and constructs a database containing information of various types of intents, i.e., builds an intent framework.

[0118] If a new intent category needs to be added, the decision cloud platform only needs to define the information included in the new intent category, so that the electronic device can recognize the new intent category. In the slot of the intent information, fill in the specific value to get a concrete intent.

[0119] 2. Mapping relationship between entity category and intent category

[0120] Entity information can include entity category and entity value. Entity information can be implemented in the following format: "entity category": ["entity value"]. The following examples illustrate several entities:

[0121] "code": ["password code"]

[0122] "location": ["address"]

[0123] "video": ["movie name"]

[0124] "time": ["date"]

[0125] "commodity": ["commodity name"]

[0126] "delicacy": ["delicacy name"]

[0127] "multiLocation": ["address 1", "address 2"]

[0128] The entity value in the above entity is replaced by the type of value, and the actual entity value needs to be filled with a concrete numerical value.

[0129] The mapping relationship of the entity category to the intent category can be predefined by the decision cloud platform.

[0130] 3. Service ranking rule in the cold start stage

[0131] The cold start stage refers to a stage before the electronic device side learns the user preference. The service ranking rule in the cold start stage can be set according to the group preference, for example, the preference order of the group for each service can be: a ride-hailing service, a collection service, a shopping service, and the like.

[0132] The above information configured by the decision cloud platform can be issued to the electronic device, for example, the decision cloud platform can share the configured information of multiple intent categories, the mapping relationship of the entity category to the mapping category to the intent recognition module, and share the service ranking rule in the cold start stage to the service ranking module.

[0133] The wisdom service platform is used to open to the third party providing the service, to facilitate the third party to define or edit the information of the service provided. The information of the service provided by the third party obtained by the wisdom service platform can be referred to as a global service system.

[0134] The information of the service defined by the third party can include one or more of the following: a service ID, a service name, an icon of the service, a service type, an identification of an intent category supported by the service, a slot name of the service, an application package name, a component name, a uniform resource locator (URL) for jumping to the service, and a parameter name. The service type refers to the form of providing the service, for example, can include a widget, a deep link, an Android shared content (share intent), a non-Android applet / light application (light APP) / HyperText Markup Language (HTML) web page, and the like. The intent supported by the service refers to the intent category that can be met by the service, which can be selected by the third party from the intent categories defined by the decision cloud platform, and the number of the item can be one or more. The slot name of the service is used to indicate the slot used to obtain the parameter from the intent when starting the service, which can be selected by the third party from the slot names of the intent category supported by the service defined by the decision cloud platform, and the number of the item can be one or more. The application package name refers to the package name of the application to which the service belongs. The component name refers to the name of the component to which the service belongs. The parameter name refers to the parameter used to pass the value filled in the slot represented by the slot name when the service is executed. The above several items are not all necessary, and some of them can be reduced according to actual needs.

[0135] If there is a newly added service that needs to access the electronic device, the smart service platform can flexibly extend the newly added service. The service provider can register the newly added service to the service management framework according to the standard format required by the smart service platform, and the electronic device can subsequently start the new service. In this way, the service provider does not need to adaptively modify the service code, but only needs to register the service information according to the standard format, so that the service can be accessed.

[0136] After obtaining the service information defined by each third party, the smart service platform can share the service information with the service management framework of the electronic device.

[0137] Based on the foregoing introduction, the smart service platform standardizes the access form of the service, and generates a database containing various service information according to the services accessed by each party. It is equivalent to mapping the differentiated execution interfaces and parameters of different services (including system services and third-party services) to the business side through the self-defined rules provided by the embodiments of the present application, so that the business side can run each service through a unified standard interface.

[0138] The functions of each module in the above framework will be explained below in combination with the specific process of the recommended service.

[0139] As shown in FIG. 6, the process of the recommended service provided by the embodiments of the present application can include the following steps:

[0140] Step S1. The first business subscribes to the intent category and / or service.

[0141] The first business referred to herein refers to a business with the function of recommending a service, for example, a smart flow service in the electronic device, a business through copying a recommended service, a business through dragging a recommended service, etc.

[0142] The first business can initiate a subscription to the service subscription module, with the purpose of subscribing to the range of intent categories and / or the range of services required by the first business. The intent category and / or service subscribed by the first business can be one or more. After the first business is started, the electronic device can identify the intent category within the range of intent categories subscribed by the first business, and can also recommend the service within the range of services subscribed by the first business according to the identified intent.

[0143] Different first businesses can subscribe to different ranges of intent categories and / or service ranges to meet the individual needs of different business parties. The first business can dynamically subscribe to the range of intents and the range of services according to its own needs, that is, to support the dynamic expansion of the intent categories and / or services supported by the first business. The range of intent categories and the range of services subscribed by the first business can be determined by the developer of the electronic device.

[0144] The service scope and intent category scope of a certain first service subscription are exemplarily shown in the following list form:

[0145] The service list of the subscription:

[0146] ["taxi service", "navigation service",

[0147] "search service", "global favorite service"]

[0148] The intent category list of the subscription:

[0149] ["social.SHARE",

[0150] "email.SEND",

[0151] "office_software.PRINT",

[0152] "note.TAKE",

[0153] "HAIL_TAXI",

[0154] "favorites.ADD",

[0155] "schedu le.CREATE",

[0156] "video.WATCH",]

[0157] The service list above contains the codes of the services of the subscription, indicating that the services of the subscription include the following: taxi service, navigation service, search service, global favorite service.

[0158] The intent category list above contains the codes of the intent categories of the subscription, indicating that the intent categories of the subscription include the following: social sharing class intent, email sharing class intent, printing class intent, note class intent, taxi class intent, favorite class intent, creating schedule class intent, watching video class intent, etc.

[0159] After the first service subscription intent category and service, if the electronic device starts the first service, the multimodal recommendation service in FIG. 6 will load the required resources in advance according to the intent category and service of the first service subscription, such as the information of the intent category of the subscription. The multimodal recommendation service can also notify the first service side to load the static resources such as the icon of the subscribed service in advance. In this way, the efficiency of service recommendation can be accelerated.

[0160] The purpose of step S1 is to let the first service subscription support the intent category and / or the service, so that the electronic device finds the corresponding intent category from the intent category of the first service subscription in subsequent S4, and facilitates the electronic device to query the corresponding service from the service range of the first service subscription in subsequent S5. Therefore, S1 can be executed before S4 and S5, and the execution time of S1 is not limited in the present application.

[0161] Step S1 is an optional step, and in some embodiments, step S1 can also not be executed. For example, all intent categories and / or all services supported by the electronic device can be defaulted, and the respective services do not need to be actively subscribed.

[0162] Step S2. The electronic device receives the first information input by the user to the first service.

[0163] The electronic device can receive the first information input by the user through the portal provided by the first service.

[0164] S2 can be executed in the process of starting the "intelligent flow transfer" service of the electronic device. S2 can also be executed in the process of starting other services other than "intelligent flow transfer" of the electronic device.

[0165] Step S3. The electronic device calls a multi-modal content understanding module in the multi-modal recommendation service to convert the received content into a first entity.

[0166] The multi-modal content understanding module can call an entity recognition capability to convert the received content into a first entity. For different types of input content, the electronic device can call different entity recognition capabilities to extract the features of the content and convert them into entities, for example, for text, natural language processing (NLP) can be used to extract its features; for pictures, computer vision (CV), optical character recognition (OCR), etc. capabilities can be used to extract their features. The entity recognition capability can include the capability of the electronic device locally, and also include the recognition capability of the cloud side. The entity recognition capability can be provided based on the multi-path recognition framework of the electronic device.

[0167] If there is a new entity recognition capability, the entity recognition capability of the electronic device can be extended through a plug-in method, so that the electronic device can use the new entity recognition capability to recognize entities.

[0168] The entity of the present application can be expressed in a standard format, that is, including an entity category and an entity value.

[0169] Step S3 corresponds to the electronic device obtaining the category and value of the entity according to the received content. For the definition of the entity category and the entity value, reference can be made to the relevant description hereinbefore. In the embodiments of the present application, the entity category can be selected within the range of entity categories supported by the electronic device. That is, the entity converted by the electronic device belongs to an entity category within the preset range.

[0170] The first entity converted by the electronic device can be one or more, and the categories of the plurality of first entities can be different.

[0171] If there is a newly added modality, the multi-modal content understanding module of the multi-modal recommendation service in FIG. 6 can add a plug-in that defines information of the newly added modality. In this way, new modalities are quickly extended through the plug-in mode, so that the electronic device can analyze the content of the newly added modality.

[0172] Step S4. The electronic device calls the intent recognition module in the multi-modal recommendation service to identify a first intent corresponding to the first entity. The first intent is a concrete intent.

[0173] Specifically, the electronic device can first find an intent category corresponding to the first entity in the intent categories defined in the decision cloud platform according to the mapping relationship between the entity category and the intent category, which can be referred to as a first intent category for example. The mapping relationship between the entity and the intent can be referred to as the introduction hereinbefore. Then, the electronic device can fill in specific values in the slot of the first intent category, so as to obtain the concrete first intent. The values filled in the slot can be the values of the first entity, or can be obtained by the electronic device through positioning, learning, and other ways.

[0174] The first intent can be limited to a valid intent, that is, the electronic device identifies a valid intent corresponding to the first entity.

[0175] In some embodiments, if step S1 is performed, the electronic device can find an intent category corresponding to the first entity in the intent categories of the first service subscription, and further find a concrete first intent.

[0176] In some embodiments, the electronic device can also find a corresponding first intent category according to the modality of the first information. The correspondence between the modality and the intent category is usually set according to the habits of a group of users, for example, the text modality can correspond to a copy intent, a search intent, and a translation intent; the picture modality can correspond to a sharing to a social application intent; the video modality can correspond to a playing intent, and the like.

[0177] The number of the first intent categories corresponding to the first entity can be one or more. The number of the first intents corresponding to the first entity can be one or more.

[0178] In the foregoing S2-S4, the electronic device first acquires the input content, and then converts the input content into an entity, and then identifies the intent corresponding to the entity. In this process, the content input by the user can include multiple modalities, and the entity converted from the input content can also include multiple entities.

[0179] An important step of the identification process for multi-modal content includes the following: loading intent metadata and business rules (also referred to as information fusion rules). The intent metadata refers to information in the global intent system, such as the encoding of various intents, slots, etc. The business rules can include the mapping relationship between entity categories and / or modalities and intent categories, content in the global service system, etc. The loaded intent metadata and business rules will be used in subsequent steps, which can be referred to in the foregoing description of S3-S4 in FIG. 6.

[0180] Determine the modal type and load various entity recognition capability models. The entity recognition capability model can include but is not limited to NLP model, CV model, etc. NLP text recognition, CV image recognition. This step is actually a serial identification of entities for different modalities of content.

[0181] Step S5. The electronic device calls the service recall module in the multi-modal recommendation service, and queries the service corresponding to the first intent based on the service information provided by the service management framework. Specifically, the electronic device can search the service information provided by the service management framework, and query the service whose identification of the intent category matches the identification of the first intent category. If the identification of the intent category supported by the service is the same as the identification of the first intent category, it means that the service can meet the intent belonging to the first intent category, that is, it can meet the first intent.

[0182] In some embodiments, if step S1 is performed, that is, the first business in the electronic device subscribes to the service range, the electronic device can query the service corresponding to the first intent in the service range subscribed by the first business when performing step S5.

[0183] The number of services corresponding to the first intent can be one or multiple, which is not limited here.

[0184] Step S6. The electronic device calls the service ranking module in the multi-modal recommendation service to rank the multiple services queried.

[0185] If multiple services are queried in step S5, step S6 can be performed. The rules for the electronic device to rank the multiple services can include multiple rules, for example, the rules can be the priority of recent use, the priority of high frequency of use, the priority of high confidence (i.e., high level) of the intent category corresponding to the intent, the order of application preference of the user, etc., which is not limited here.

[0186] Step S6 is an optional step, and in some embodiments, the plurality of services can not be sorted.

[0187] Step S7. The electronic device calls the service notification module and returns the obtained information of the service to the first application.

[0188] The electronic device can return the information of the plurality of services to the first application in a list form. If there is sorting, the electronic device can return the sorted information of the services to the first application so that the first application knows the sorting. The information of the service can refer to the foregoing description. In addition, the electronic device can first determine the slot name of the service, then find the value of the same slot name in the first intent, and return the value to the first application.

[0189] For example, if the first intent is as shown in Table 2 and the service is as shown in Table 3, the electronic device can return the value (i.e., “Shenzhen Airport”) of the slot named “end_name” in the first intent to the first application together with the information of the service.

[0190] Step S8. The electronic device displays the icon of the service obtained in step S7 on the user interface provided by the first application.

[0191] If the sorting information is obtained in step S7, the electronic device can display the icons of the services obtained in step S7 from top to bottom according to the sorting.

[0192] Step S9. The electronic device receives a user operation on the icon of the first service, and in response to the user operation, the first application shares the first information to the first service.

[0193] Referring to step S7, the first application of the electronic device can start the service according to the information of the service, for example, find the application package name of the service to start the application to which the service belongs, then jump to the service according to the jump URL of the service, and then pass the slot value of the service obtained to the parameter specified by the service, so as to share the first information to the service.

[0194] For example, if the user selects the taxi service of the map application A, the electronic device can pass the address “xx City, xx District, xx Road, xx Building, xx” input by the user to the taxi service through the “keyword” parameter. After the electronic device starts the taxi service, a taxi page with the end point filled with the address is displayed.

[0195] From the foregoing description, the functions of the modules on the electronic device side in the framework shown in FIG. 6 can be known.

[0196] The conflict resolution method provided in the embodiments of the present application can be executed in a mobile phone with the above-mentioned hardware and software structures.

[0197] The following takes several specific scenarios as examples to illustrate the specific implementation of conflict identification and response.

[0198] Scenario one, video scenario.

[0199] Video scenario refers to the scenario in which the foreground displays a video playback window.

[0200] Referring to FIG. 7, after the mobile phone runs the video player in the foreground, it can display interface 701, which includes the covers of multiple videos, such as cover 7011, cover 7012, etc. Interface 701 does not include a video playback window and does not belong to a video scenario. Subsequently, in response to the user's triggering operation on any cover in interface 701, the video corresponding to the cover can be played in the video playback window, which then belongs to a video scenario. Taking any cover as cover 7011 in interface 701 as an example, in response to the user's triggering operation on cover 7011 in interface 701, the mobile phone can display interface 702, which includes video playback window 7021, in which the video corresponding to cover 7011 in interface 701, such as video A, is being played.

[0201] It should be noted that the video playback window can play or pause the video, and accordingly, the video scenario can further include the scenario in which the video is played in the video playback window (such as the scenario shown in interface 702) and the scenario in which the video is paused in the video playback window.

[0202] Of course, in addition to the above-mentioned video player, the mobile phone can also display a video playback window for playing a video during the running of some other applications. For example, the mobile phone runs a browser in the foreground and displays a video playback window in the application interface of the browser for playing a video. For another example, the mobile phone runs a chat application in the foreground and displays a video playback window in full screen in the video panel (such as the video number panel in WeChat TM ) of the chat application for playing a video.

[0203] In the video scenario, performing a long press operation on the video being played can trigger fast forward. That is, the other functions that conflict with the arbitrary gate include the fast forward function. Based on this, in the video scenario, in response to the long press operation on the video being played, the mobile phone can fast forward the video without responding to the arbitrary gate, thereby ensuring the normal response of the fast forward function.

[0204] Continuing to refer to FIG. 7, in response to the user's long press operation on video playback window 7021 in the above-mentioned interface 702, the mobile phone can display interface 703, in which video playback window 7021 displays a prompt of "fast forwarding" 7031, indicating that it is fast forwarding, i.e., normally responding to the fast forward function.

[0205] In a specific implementation, when playing a video in the gallery application, the gallery application does not have a special response to the long press operation of the user. That is, when playing a video in the gallery application, there is usually no function that conflicts with the arbitrary door, and accordingly, the video scene can be excluded from the scene of playing a video in the gallery application.

[0206] Of course, performing a long press operation on a playing video can also trigger functions other than fast forwarding, such as a speed function, a sharing function, etc., which are not limited in the present application.

[0207] In the video scene, in addition to the case of performing a long press operation on a playing video, the mobile phone can respond to the arbitrary door in the following case 1 and case 2.

[0208] Case 1: performing a long press operation on an area other than a video.

[0209] Referring to FIG. 8, in response to the long press and drag operation of the user on the text area 7022 of the interface 702, the mobile phone can display an interface 801, which includes shortcut entrances of multiple services matched with the content of the text area 7022, that is, responds to the arbitrary door.

[0210] It should be noted that when detecting the long press operation of the user on the text area 7022 of the interface 702 shown in FIG. 8, the mobile phone can identify that it is not a long press operation on a playing video, there is no conflict, and therefore no response until detecting the drag operation on the text area 7022. The mobile phone then responds to the arbitrary door.

[0211] Case 2: performing a long press operation on a paused video. It should be noted that pausing the video can include two cases: pausing the video in response to the pause operation of the user, or pausing the video due to network reasons, etc. This paper mainly takes pausing the video in response to the pause operation of the user as an example.

[0212] Continuing to refer to FIG. 8, in response to the pause operation of the user on the video playing window 7021 in the interface 702, such as a double-click operation, the mobile phone can display an interface 802, in which the video in the video playing window 7021 is in a paused state. In response to the long press and drag operation of the user on the video playing window 7021 in the interface 802, the mobile phone can display an interface 803. The interface 803 includes shortcut entrances of multiple services matched with the video in the video playing window 7021, that is, responds to the arbitrary door.

[0213] It should be noted that, when detecting the long-press operation of the user on the video playing window 7021 in the interface 802 shown in FIG. 8, the phone can recognize that the long-press operation is not on the video being played, there is no conflict, and thus no response is made until detecting the drag operation of the user on the video playing window 7021 in the interface 802, and the phone responds to any gate.

[0214] In the above case 2, in response to the long-press and drag operation of the user on the video playing window 7021 in the interface 802, the phone can recognize the intention of the user based on the current paused video frame in the video playing window 7021, and respond with the current paused video frame as the object of the drag (as shown in the picture 8031 in the interface 803).

[0215] In some embodiments, in the video scene, the phone can recognize whether there is a conflict by the following conflict condition 1, i.e., recognize whether the long-press operation is performed on the video being played.

[0216] The conflict condition 1 includes the following condition content: (1) the video is being played; (2) the long-press operation is on the video playing window. If the above (1) and (2) are met, it indicates that the user is performing a long-press operation on the video being played.

[0217] In a specific implementation, the phone can detect whether the video is being played by sensing whether the video decoding is being performed, i.e., detecting whether the above (1) is met. The phone can obtain each video frame by video decoding and send it to the display screen for display, thereby realizing video playing. Based on this, if it is sensed that the video decoding is being performed, the phone can determine that the video is being played. If it is not sensed that the video decoding is being performed, the phone can determine that the video is not being played.

[0218] Among them, the video decoding includes two ways of soft decoding and hard decoding. Hard decoding is to decode the high-definition video by the GPU of the phone, and its advantages include low CPU occupancy, smooth playing, and low power consumption, and its disadvantages include being limited by video format, needing to adapt and debug the decoder, and poor stability. Soft decoding is to decode by the CPU of the phone, and its advantages include not being limited by video format and better stability, and its disadvantages include higher power consumption, higher resource occupancy, and not as smooth as hard decoding for high-definition video.

[0219] In practice, the same application can use the same video decoding mode, or different decoding modes for different videos. Different applications can use the same video decoding mode, or different video decoding modes. These cases cannot be predicted in advance. Based on this, the mobile phone can perceive whether hard decoding is performed, and whether soft decoding is performed. And in the case of perceiving that hard decoding is performed or soft decoding is performed, the mobile phone can determine that the video is playing. In the case of neither perceiving hard decoding nor perceiving soft decoding, the mobile phone can determine that the video is not playing.

[0220] For example, the mobile phone can perceive hard decoding through a mediaCodec, and can perceive soft decoding through a SurfaceFlinger. If neither hard decoding nor soft decoding is performed, neither the mediaCodec nor the SurfaceFlinger can perceive the result, such as returning false.

[0221] It should be noted that the mobile phone can also use other ways to detect whether the video is playing. For example, the mobile phone can also take screenshots of the video playing window multiple times, compare the screenshot contents, if the difference between the screenshot contents is large, it indicates that the video is playing, and if the difference between the screenshot contents is small, it indicates that the video is not playing.

[0222] In a specific implementation, the mobile phone can detect whether the long press operation is directed to the video playing window by the view type corresponding to the region pressed by the long press operation, that is, to detect whether (2) above is met. Generally, the mobile phone can display the video in a VideoView, a SurfaceView, and a TextureView. Based on this, the mobile phone can detect whether the view type corresponding to the region pressed by the long press operation is a preset type, such as the VideoView, the SurfaceView, and the TextureView. If the view type corresponding to the region pressed by the long press operation is the preset type, the mobile phone can determine that the long press operation is directed to the video playing window. If the view type corresponding to the region pressed by the long press operation is not the preset type, the mobile phone can determine that the long press operation is not directed to the video playing window.

[0223] It should be noted that the mobile phone can also use other ways to detect whether the long press operation is directed to the video playing window. For example, the mobile phone can also compare the pressed region of the long press operation with the region of the video playing window. If the pressed region is within the region of the video playing window, it indicates that the long press operation is directed to the video playing window. If the pressed region is not within the region of the video playing window, it indicates that the long press operation is not directed to the video playing window.

[0224] If the conflict condition 1 is satisfied, it indicates that the long press operation is performed on the video being played, and there is a conflict. If the conflict condition 1 is not satisfied, it indicates that the long press operation is not performed on the video being played, and there is no conflict. Wherein, the conflict condition 1 is not satisfied includes that (1) and / or (2) above are not satisfied. For example, the aforementioned case 1 satisfies (1) and does not satisfy (2), and the case 2 satisfies (2) and does not satisfy (1).

[0225] In the above description of the video scene, the mobile phone mainly displays the application interface of a single application in the foreground, such as the interface of the video player.

[0226] In practice, the mobile phone can also display the application interfaces of multiple applications in the foreground at the same time, wherein the simultaneous display can be a split-screen display; or the simultaneous display can be a full-screen display, and the remaining small windows (floating windows) are displayed. The present application does not make specific limitations thereto.

[0227] When multiple application interfaces of applications are displayed in the foreground at the same time, a video can be playing in the application interface of one of the applications. It should be noted that at the same time, only one application is usually allowed to play a video, and two typical cases can be seen in the interface 901 and the interface 902 in FIG. 9.

[0228] Referring to the interface 901 in FIG. 9, the mobile phone displays the application interface 9011 (such as the same as the interface 702 in the foregoing) and the application interface 9012 (such as the same as the interface 201 in the foregoing) in the foreground in a split-screen mode. The application interface 9011 includes the video playing window 9013, and a video is playing in the video playing window 9013. The application interface 9012 does not include a video playing window, and no video is playing.

[0229] Referring to the interface 902 in FIG. 9, the mobile phone displays the application interface 9021 (such as the same as the interface 702 in the foregoing) and the application interface 9022 in the foreground. The application interface 9021 includes the video playing window 9023, and a video is playing in the video playing window 9023. The application interface 9022 includes the video playing window 9024, but the video playing window 9024 has paused playing a video.

[0230] For the above similar cases in FIG. 9, as in the foregoing video scene, in response to the long press operation on the video being played, such as the long press operation on the video playing window 9013 in the interface 901 or the long press operation on the video playing window 9023 in the interface 902, the mobile phone can fast forward the video. In addition, as in the following case 3 and case 4, the mobile phone can respond to any door.

[0231] Case 3: The long press operation on the application interface without the video playing window.

[0232] Continuing to refer to FIG. 9, in response to the user's long-pressing and dragging operation on the application interface 9012 in the interface 901, the phone can display the interface 903, which includes shortcut entrances of multiple services matching the content (such as the short message content) in the application interface 9012, i.e., responding to the arbitrary gate.

[0233] It should be noted that when detecting the user's long-pressing operation on the application interface 9012 in the interface 901 shown in FIG. 9, the phone can identify that the long-pressing operation is not on the playing video, and there is no conflict, and thus no response is made until detecting the dragging operation on the application interface 9012, and the phone responds to the arbitrary gate.

[0234] Case 4, long-pressing operation on the video playing window in which the video is paused.

[0235] Continuing to refer to FIG. 9, in response to the user's long-pressing and dragging operation on the video playing window 9024 in the interface 902, the phone can display the interface 904, which includes shortcut entrances of multiple services matching the content in the video playing window 9024, i.e., responding to the arbitrary gate.

[0236] It should be noted that when detecting the user's long-pressing operation on the video playing window 9024 in the interface 902 shown in FIG. 9, the phone can identify that the long-pressing operation is not on the playing video, and there is no conflict, and thus no response is made until detecting the dragging operation on the video playing window 9024, and the phone responds to the arbitrary gate.

[0237] Corresponding to the case shown in FIG. 9 described above, the conflict condition 1 described above can further include: (3) the application to which the long-pressing operation is directed and the application in which the video is playing are the same.

[0238] In the case of simultaneously satisfying (1), (2), and (3), the phone can identify that there is a conflict.

[0239] Taking the long-pressing operation on the video playing window 9013 in the interface 901 described above as an example: the video playing window 9013 in the application interface 9011 is playing a video, satisfying (1); the long-pressing operation is directed to the video playing window 9013, satisfying (2); the application to which the long-pressing operation is directed and the application in which the video is playing are the same, satisfying (3). The phone can identify that there is a conflict, and can fast forward the video.

[0240] Taking a long press operation on the video playing window 9023 in the interface 902 as an example: the video playing window 9023 in the application interface 9021 is playing a video, satisfying (1); the long press operation is directed to the video playing window 9023, satisfying (2); the application to which the long press operation is directed and the application that is playing the video are the same, satisfying (3). The phone can recognize that there is a conflict and can fast forward the video.

[0241] Conversely, any one of (1), (2) and (3) is not satisfied, and the phone can recognize that there is no conflict.

[0242] Taking the above case 3 as an example: the video playing window 9013 in the application interface 9011 is playing a video, satisfying (1); the long press operation is not directed to the video playing window, not satisfying (2); the application to which the long press operation is directed and the application that is playing the video are not the same, not satisfying (3). Therefore, for case 3, the phone can recognize that there is no conflict and can respond to any door.

[0243] Taking the above case 4 as an example: the video playing window 9023 in the application interface 9021 is playing a video, satisfying (1); the long press operation is directed to the video playing window 9024, satisfying (2); the application to which the long press operation is directed and the application that is playing the video are not the same, not satisfying (3). Therefore, for case 4, the phone can recognize that there is no conflict and can respond to any door.

[0244] As can be seen, in the above case 4, combined with (3), in the scenario of running multiple applications in the foreground, the conflict can be more accurately identified.

[0245] Scenario two, recording / video recording scenario.

[0246] The recording scenario refers to a scenario in which a microphone is being used. For example, a scenario of recording in a recording application, a scenario of sending a voice message in a chat application, etc.

[0247] The video recording scenario refers to a scenario in which a camera is being used. For example, a scenario of running a camera application in the foreground, a scanning scenario, etc.

[0248] In the recording / video recording scenario, the user performs a long press operation, which can trigger recording / video recording related control functions, such as starting recording, ending recording, starting shooting, ending shooting, etc. That is, other functions that conflict with any door include recording / video recording related control functions. Based on this, in the recording / video recording scenario, in response to the user's long press operation in the recording / video recording interface, the phone can perform the above control functions without responding to any door, thereby ensuring the normal response of the control function.

[0249] Taking recording in a recording application as an example, referring to FIG. 10, the mobile phone can display interface 1001, which is a recording interface of the recording application, and recording is being performed at this time. In response to a long press operation of the user on recording control 10011 in interface 1001, the mobile phone can display interface 1002, in which recording has ended, i.e., the function of ending recording is normally responded to.

[0250] It should be noted that if the long press operation is not directed at a recording control / video recording control, it can not be responded to, and in general, any gate will not be responded to.

[0251] In practice, the mobile phone can also record in the background. Taking recording as an example, after starting recording, the mobile phone can switch the recording application to the background, thereby enabling recording in the background. When recording / video recording in the background, the interface displayed in the foreground of the mobile phone can not be a recording / video recording interface (abbreviated as a non-recording / video recording interface), and performing a long press operation in the non-recording / video recording interface will not conflict with the control function related to recording / video recording. Based on this, in the recording / video recording scenario, in response to a long press operation of the user in the non-recording / video recording interface, the mobile phone can respond to any gate, thereby ensuring that the function of any gate is provided to the user without conflict.

[0252] Referring to FIG. 11, in response to a sliding operation of the user from the bottom of interface 1001 upwards, the mobile phone can display interface 1101 (e.g., the same as interface 202 described above). Interface 1101 is the desktop of the mobile phone, and the top status bar of interface 1101 includes a recording status prompt 11011, i.e., recording in the background. In response to a triggering operation of the user on application icon 11012 of the short message application in interface 1101, the mobile phone can display interface 1102 (e.g., the same as interface 201 described above), which is an application interface of the short message application, and interface 1102 is a non-recording / video recording interface. The difference from interface 201 described above is that the top status bar of interface 1102 also includes recording status prompt 11011, indicating that recording in the background is still being performed. In response to a long press and drag operation of the user on short message 11021 in interface 1102, the mobile phone can display interface 1103, which includes shortcut entrances of multiple services matched with the content of short message 11021, i.e., any gate is responded to.

[0253] It should be noted that when detecting the long press operation of the user on short message 11021 in interface 1102 shown in FIG. 11, the mobile phone can identify that it is a long press operation on a non-recording / video recording interface, and there is no conflict, so it is not responded to until the drag operation on short message 11021 is detected, and the mobile phone responds to any gate.

[0254] Of course, in the case that the application interface of multiple applications is displayed in the foreground, and one of the applications is recording audio / video, the mobile phone can also respond to any door in response to the long-press and drag operation of the user on the application interface of the application without recording audio / video. In other words, in the recording audio / video scenario, the mobile phone can respond to any door in response to the long-press and drag operation on the non-recording audio / video interface.

[0255] In some embodiments, the mobile phone can identify whether there is a conflict by the following conflict condition 2, i.e., identify whether the long-press operation is performed on the recording audio / video interface. If the above conflict condition 2 is met, there is a conflict. If the above conflict condition 2 is not met, there is no conflict.

[0256] The conflict condition 2 includes the following condition content: (A) the microphone or camera has been started; (B) the application to which the long-press operation is directed and the application that invokes the microphone or camera are the same. If the above (A) and (B) are met, it indicates that the user performs the long-press operation on the recording audio / video interface, and there is a conflict. If (A) or (B) is not met, it indicates that the user does not perform the long-press operation on the recording audio / video interface.

[0257] Taking the long-press operation on the recording control 10011 in the above interface 1001 as an example, the current recording is being performed, (A) is met; the application to which the long-press operation is directed is the recording application that is currently recording, (B) is met. The mobile phone can identify that there is a conflict and can end the recording.

[0258] Taking the long-press operation on the short message 11021 in the above interface 1102 as an example, the current recording is being performed in the background, (A) is met; the short message application to which the long-press operation is directed and the recording application that invokes the microphone are not the same application, (B) is not met. The mobile phone can identify that there is no conflict and can respond to any door.

[0259] In a specific implementation, the mobile phone can monitor the start and stop of the microphone or camera to detect whether the microphone or camera is started, i.e., to detect whether the above (A) is met.

[0260] In a specific implementation, the mobile phone can intercept the invocation event of the microphone or camera and parse the invocation information therefrom to obtain and record the application that invokes the microphone or camera. Subsequently, after receiving the long-press operation, the mobile phone can match the application to which the long-press operation is directed with the recorded application. If the recorded application includes the application to which the long-press operation is directed, the mobile phone can detect that the application to which the long-press operation is directed and the application that invokes the microphone or camera are the same, i.e., detect that the above (B) is met. If the recorded application does not include the application to which the long-press operation is directed, the mobile phone can detect that the application to which the long-press operation is directed and the application that invokes the microphone or camera are not the same, i.e., detect that the above (B) is not met.

[0261] Scenario three, the control itself can be dragged.

[0262] Some controls in the interface displayed by the phone support drag response, such as deleting by dragging, adjusting order by dragging, etc. That is, other functions corresponding to the drag response supported by these controls conflict with the response of any door. Therefore, if the phone performs a long press and drag operation on these controls, the response of any door is in conflict, and the phone can perform the drag response of the control itself instead of the response of any door, thereby avoiding the conflict.

[0263] For example, performing the drag response of the control itself includes moving the control with the drag, and after the displacement of the control exceeds a displacement threshold, such as exceeding 50 pixels, performing operations such as deleting, adjusting order, etc. Different controls that support drag response may have different displacement thresholds.

[0264] Referring to FIG. 12, the phone can display interface 1201. Interface 1201 is an interface for publishing social dynamics, and in interface 1201, social dynamics can be edited. For example, the text "AAA…" 12011 is input in interface 1201, and an image 12012 is added, etc. Image 12012 is an operable control, and the control supports a drag response to delete the image. Specifically, in response to a user's long press and drag operation on image 12012, the phone can display interface 1202. Interface 1202 includes a hot zone "Drag here to delete" 12021, which can be used to trigger the deletion of image 12012. Subsequently, in response to the user dragging image 12012 to "Drag here to delete" 12021, the phone can delete image 12012, and after deletion, interface 1203 can be displayed, which no longer includes image 12012. Therefore, for the control of image 12012, after detecting a long press and drag operation on the control, the phone can not respond to any door, but perform the original delete response, such as displaying the hot zone "Drag here to delete" 12021, deleting image 12012, etc.

[0265] In some embodiments, the phone can identify whether there is a conflict through conflict condition 3, that is, whether a long press and drag operation is performed on a control that supports a drag response.

[0266] Conflict condition 3 can include the following condition content: (a) the long press operation is on a control that registers a drag listener (DragListener). If (a) is met, it indicates that the long press operation is on a control that supports a drag response. If (a) is not met, it indicates that the long press operation is not on a control that supports a drag response.

[0267] Generally, the phone can register a DragListener for the control, so that the control can support the drag response. For example, the phone can register a DragListener for the control corresponding to the image 12012 in FIG. 12, and the control corresponding to the image 12012 supports the drag response of deleting the image only after the DragListener is registered.

[0268] In a specific implementation, the phone can maintain a list of controls registered with the DragListener, and compare the control pressed by the long press operation with the list of controls. If the control targeted by the long press operation is in the list of controls, it indicates that the long press operation is targeted at the control registered with the DragListener. If the control targeted by the long press operation is not in the list of controls, it indicates that the long press operation is not targeted at the control registered with the DragListener. Thus, the detection of (a) above is completed.

[0269] It should be noted that the phone can also use other ways to detect whether the long press operation is targeted at the control registered with the DragListener. For example, the phone can also compare the pressed area of the long press operation with the area where the control registered with the DragListener is located. If the pressed area is in the area where the control registered with the DragListener is located, it indicates that the long press operation is targeted at the control registered with the DragListener. If the pressed area is not in the area where the control registered with the DragListener is located, it indicates that the long press operation is not targeted at the control registered with the DragListener.

[0270] Scenario four, game scenario.

[0271] The game scenario refers to a scenario in which the phone runs a game in the foreground. It should be understood that the game can be a small program game, a web game, an independent application game, etc., which is not limited in the present application.

[0272] In the game scenario, the user's long press operation can trigger a game operation, such as accelerating forward, continuous shooting, etc. That is, the other functions in conflict with the arbitrary gate include the functions of the game operation. Based on this, in the game scenario, the phone can respond to the corresponding game operation in response to the long press operation, without performing the response of the arbitrary gate, thereby avoiding the conflict.

[0273] Referring to FIG. 13, after starting the game application, the mobile phone can display interface 1301, which includes game screen 13011. In response to a long press operation of the user on the forward button 13012 in the game screen 13011, the mobile phone can display interface 1302, in which the character 13021 in the game screen 13011 is accelerating forward (indicated by symbol 13022 in the figure), that is, responding to the skill of accelerating forward.

[0274] In some embodiments, the mobile phone can identify whether there is a conflict by the following conflict condition 4, that is, identify whether it is a long press operation on the game. If the conflict condition 4 is met, there is a conflict, and if the conflict condition 4 is not met, there is no conflict.

[0275] The conflict condition 4 includes at least one of the following condition contents: ① the application running in the foreground of the mobile phone is a game application; ② the current top activity of the mobile phone is in the activity blacklist; ③ the sound feature and / or the screen feature of the game scene are met. The following will be described respectively:

[0276] Regarding ①

[0277] The mobile phone is running a game application in the foreground, and the long press operation of the user is on the game screen.

[0278] Further, the mobile phone can detect whether the application running in the foreground is a game application by the following way:

[0279] Method one, the mobile phone can detect whether the application running in the foreground is a game application based on the attribute value of the application.

[0280] In the attribute of the application, there is an attribute value of whether the application is a game application, and the mobile phone can obtain the attribute value through a preset interface. The preset interface can be ApplicationInfo.FLAG_IS_GAME or ApplicationInfo.CATEGORY_GAME. If it is a game application, the return value true can be obtained through the preset interface. If it is not a game application, the return value false can be obtained through the preset interface. Thus, whether it is a game application can be detected.

[0281] Method two, the mobile phone can match the application running in the foreground with the application list of the game application in the application market (or application vendor), and detect whether the application running in the foreground is a game application.

[0282] The application market includes various categories of applications, such as audio and video, shopping, social, game, and the like. The game category includes a large number of game applications. If the foreground running application (such as an application identifier) is in the application list of game applications in the application market, the phone can detect that the foreground running application is a game application. If the foreground running application is not in the application list of game applications in the application market, the phone can detect that the foreground running application is not a game application.

[0283] In the third way, the phone can match the foreground running application with the game list in the game manager application to detect whether the foreground running application is a game application.

[0284] If the foreground running application (such as an application identifier) is in the game list of the game manager application, the phone can detect that the foreground running application is a game application. If the foreground running application is not in the game list of the game manager application, the phone can detect that the foreground running application is not a game application.

[0285] Of course, in actual implementation, at least two of the above-mentioned first way, second way, and third way can be combined, and in the case where the condition of any of the at least two ways is met, it is detected that the foreground running application is a game application, otherwise it is not a game application.

[0286] Taking the combination of the first way, the second way, and the third way as an example, through the preset interface, if the return value is true, or the foreground running application is in the application list of game applications in the application market, or the foreground running application is in the game list of the game manager application, the phone can detect that the foreground running application is a game application. Through the preset interface, if the return value is false, and the foreground running application is neither in the application list of game applications in the application market nor in the game list of the game manager application, the phone can detect that the foreground running application is not a game application.

[0287] The above-mentioned ① can be mainly used for detection of independent game applications.

[0288] Regarding ②

[0289] The activity blacklist is used to record activities for displaying game pictures, such as activities of applet games and activities of web games.

[0290] If the activity at the top of the phone is in the activity blacklist, the foreground displays a game picture. In this case, the user's long press operation is for the game picture.

[0291] For example, the phone runs a chat application in the foreground, and the top activity is the chat application. Since the chat application is not in the activity blacklist, the phone can detect that the foreground display is not a game screen. Then, the phone starts a game applet, and the top activity is the game applet. Since the game applet is in the activity blacklist, the phone can detect that the foreground display is a game screen.

[0292] The above ② can be mainly used for detection of applet games and web games. In this way, the phone can accurately detect whether the foreground display is a game screen for applet games, web games, and other games that cannot perceive the game scene.

[0293] Regarding ③

[0294] The phone can detect whether the foreground is running a game by recognizing the screen features of the interface displayed in the foreground of the phone and / or recognizing the sound features through artificial intelligence (AI). When the phone recognizes that the screen features meet the game scene and / or the sound features meet the game scene, the phone can detect that the foreground is running a game. In this case, the user's long press operation is for the game screen. When the phone recognizes that neither the screen features meet the game scene nor the sound features meet the game scene, the phone can detect that the foreground is not running a game. In this case, the user's long press operation is not for the game screen.

[0295] In actual implementation, when any one of the above ①, ②, and ③ is met, the phone can recognize that the long press operation is performed on the game screen. Conversely, when none of ①, ②, and ③ is met, the phone can recognize that the long press operation is not performed on the game screen.

[0296] In the above description of the game scene, the phone displays the application interface of a single application in the foreground, such as the application interface of a game application.

[0297] In practice, the phone can also display the application interfaces of multiple applications in the foreground at the same time. Moreover, when the phone displays the application interfaces of multiple applications in the foreground at the same time, the application interface of one application may display a game screen, and the application interfaces of other applications may or may not display a game screen.

[0298] Referring to FIG. 14, the phone can display an interface 1401, which includes an application interface 14011 (e.g., the same as the interface 1301 described above) and an application interface 14012 (e.g., the same as the interface 201 described above). The application interface 14011 includes a game screen, and the application interface 14012 does not include a game screen.

[0299] For the similar situation as shown in Fig. 14, the phone can respond to the corresponding game operation in response to the long press operation on the game picture, similar to the game scene described above. With continued reference to Fig. 14, in response to the long press operation of the user on the forward button 14013 in the application interface 14011 in the interface 1401, the phone can display the interface 1402, and the character 14021 in the application interface 14011 in the interface 1402 is accelerating forward, i.e., responding to the skill of accelerating forward.

[0300] In addition, in response to the long press operation on the non-game picture, the phone can respond to the arbitrary door. It should be noted that in the scene of running a single application in the foreground, the game picture will generally cover the screen, as shown in the game picture 13011 included in the interface 1301 described above, so that the long press operation on the non-game picture will not generally occur. Therefore, the long press operation on the non-game picture mainly occurs in the scene of running multiple applications in the foreground.

[0301] With continued reference to Fig. 14, in response to the long press operation of the user on the application interface 14012 in the interface 1401, the phone can display the interface 1403. The interface 1403 includes the shortcut entry of multiple services matched with the content (such as short message) in the application interface 14012, i.e., responding to the arbitrary door.

[0302] It should be noted that in this paper, the shortcut entry of the service is mainly taken as an example to be displayed on the right edge, and in practice, it can also be displayed in other positions, such as the left edge, which is not limited in the present application.

[0303] It should be noted that when detecting the long press operation of the user on the application interface 14012 in the interface 1401 shown in Fig. 14, the phone can identify that it is not a long press operation on the game picture, and there is no conflict, so it does not respond until detecting the drag operation on the application interface 14012, and the phone responds to the arbitrary door.

[0304] Corresponding to the similar situation as shown in Fig. 14, the above conflict condition 4 can further include: ④ the long press operation is directed to the game picture. In the case of satisfying any one of the above ①, ② and ③, and satisfying ④, the phone can identify that there is a conflict.

[0305] Taking the long press operation on the application interface 14011 in the above interface 1401 as an example: the application interface 14011 is the application interface of the game application, satisfying ①; the application interface 14011 on which the long press operation is directed includes the game picture, satisfying ④. The phone can identify that there is a conflict, and can respond to the game operation.

[0306] In the case of satisfying any one of the above ①, ② and ③, but not satisfying ④, the phone can identify that there is no conflict.

[0307] Taking a long press operation on the application interface 14012 in the interface 1401 as an example: the application interface 14011 is an application interface of a game application, which meets ①; but the application interface 14012 to which the long press operation is directed does not include a game picture, which does not meet ④. The phone can recognize that there is no conflict and can respond to any door.

[0308] It can be seen that in combination with ④, for the scenario of running multiple applications in the foreground, the conflict can be more accurately recognized.

[0309] Scenario five, other scenarios.

[0310] The other scenarios can refer to all or part of the scenarios other than the above scenarios one to four.

[0311] For other scenarios, the phone can analyze the preset constituent elements in the publishing content through layout analysis technology. The preset constituent element refers to an element that can respond to a long press operation or a long press and drag operation. That is, the other functions of the arbitrary door conflict include the functions that these preset constituent elements can respond to. Based on this, in response to the user's long press operation or long press and drag operation on the preset constituent element, the phone can execute the response corresponding to the preset constituent element.

[0312] That is, in scenario five, the conflict condition (denoted as conflict condition 5) can include: the long press operation is directed to a preset constituent element.

[0313] In addition, in response to the user's long press and drag operation on a non-preset constituent element, the phone can respond to any door.

[0314] For example, the preset constituent element includes a title bar (also known as a tab bar), a selection button, etc. The following will take the title bar element and the button element as examples to illustrate the response to the long press operation or the long press and drag operation.

[0315] Title bar

[0316] The title bar includes multiple tabs, and any title tab can be used to trigger the switching of the tab page. In addition, in response to the user's long press and drag operation on the tab, the phone can also adjust the arrangement order of the multiple tabs, that is, it can not respond to any door.

[0317] Referring to Fig. 15, the phone can display an interface 1501 including a title bar 15011. The title bar 15011 includes a plurality of tabs, such as "Focus", "Recommend", "Forum", etc. The current selected one is "Recommend", and accordingly, the interface 1501 displays the content of the "Recommend" page, such as a recommended content 15012, etc. In response to a long-press-and-drag operation of the user on "Focus", the phone can adjust the position of "Focus" in the title bar 15011 based on the displacement of the drag. For example, after the adjustment, the phone can display an interface 1502. In the interface 1502, the title bar 15011 has "Focus" adjusted to be after "Recommend".

[0318] In response to a long-press-and-drag operation of the user on the content outside the title bar, the phone can recognize that there is no conflict, and thus can respond to the arbitrary gate.

[0319] Continuing to refer to Fig. 15, in response to a long-press-and-drag operation of the user on the recommended content 15012 in the interface 1501, the phone can display an interface 1503 including a plurality of shortcut entries of services matching the content in the recommended content 15012, i.e., responding to the arbitrary gate.

[0320] Selection button

[0321] Generally, the selection button can be used to trigger the selection / deselection of certain information, such as functions, preferences, parameters, etc. Moreover, in response to a long-press-and-drag operation of the user on the selection button, the phone can adjust the arrangement order among a plurality of selection buttons, i.e., can not respond to the arbitrary gate.

[0322] Referring to Fig. 16, the phone can display an interface 1601 including a plurality of selection buttons, such as "Short Video", "Video", "Picture", "Entertainment", etc. In response to a long-press-and-drag operation of the user on "Entertainment", the phone can adjust the position of "Entertainment" among the plurality of selection buttons based on the displacement of the drag. For example, after the adjustment, the phone can display an interface 1602. In the interface 1602, "Entertainment" is adjusted to be the first among the plurality of selection buttons.

[0323] It is worth noting that in the foregoing introduction to FIG. 1, it is mentioned that for the drag operation of the operation button (which can also be understood as a control), the mobile phone can not respond to any door. However, the non-response here has a prerequisite: the mobile phone can identify that the press is an operation button. If the mobile phone cannot identify that the press is an operation button, it may still respond to any door, resulting in a conflict. With the solutions of scenarios one to five, for the long press operation of some specific controls, such as the controls in the video, the controls in the audio / video interface, the controls that can be dragged by themselves, the controls in the game screen, and the controls included in the preset constituent elements (such as the selection button), the mobile phone will not respond to the drag operation of any door, which can further compensate for the defect that a conflict is caused due to the inability to identify the control.

[0324] Taking the control in the game screen as an example, the mobile phone may have difficulty identifying the control, but with the solution of scenario four, the long press operation of the game screen is not responded to by any door, so a conflict will not be caused due to the inability to identify the control in the game screen.

[0325] At this point, it should be noted that in actual implementation, the mobile phone can identify whether a conflict exists based on one or more conflict conditions in scenarios one to five, and if a conflict is identified, the mobile phone will not respond to any door, and if a conflict is not identified, the mobile phone can respond to any door.

[0326] Referring to FIG. 17, in a specific implementation, the conflict resolution method includes the following steps:

[0327] S1701, after receiving the long press operation of the user, the mobile phone detects whether a conflict condition is met. If so, S1702 is performed; if not, S1703 is performed.

[0328] Taking the conflict conditions as including the foregoing conflict condition 1 to conflict condition 5 as an example, referring to FIG. 18, the condition that meets the conflict condition includes any one of conflict condition 1 to conflict condition 5. For conflict condition 1 to conflict condition 5, refer to the foregoing description, which will not be repeated here.

[0329] S1702, the mobile phone responds to other functions that conflict with any door.

[0330] Continuing to refer to FIG. 18, if conflict condition 1 is met, the mobile phone can respond to the video operation, such as fast forwarding the video; if conflict condition 2 is met, the mobile phone can respond to the audio / video related control function, such as ending the audio recording; if conflict condition 3 is met, the mobile phone can perform the drag response of the control itself, such as deleting the control; if conflict condition 4 is met, the mobile phone can respond to the game operation, such as fast forwarding; and if condition 5 is met, the mobile phone can perform the drag response of the preset constituent element itself, such as adjusting the order.

[0331] In this way, the mobile phone can ensure normal response of other functions in the case of identifying the existence of conflict.

[0332] S1703, the mobile phone responds to the dragging operation of the arbitrary door.

[0333] For example, in response to the dragging operation, the mobile phone can provide a shortcut entry of a service matching the dragged content, and in response to the release after dragging to the position of any shortcut entry, the mobile phone can run the corresponding application in the foreground, thereby providing the user with a shortcut service.

[0334] In this way, the mobile phone can respond to the arbitrary door in the case of identifying the non-existence of conflict, thereby realizing the direct access of a service across applications.

[0335] It should be noted that if there is no continuous dragging operation after the long pressing, the mobile phone will not respond to the arbitrary door.

[0336] In some embodiments, before S1703, such as after receiving the long pressing operation and before detecting whether the conflict condition is met, or after detecting that the conflict condition is not met and before performing S1703, the mobile phone can identify the category of the pressed object. If it is text or a picture (a paused video frame also belongs to a picture), the mobile phone performs S1703. If it is a blank area or an operation button, the mobile phone does not perform S1703 even if the conflict condition is not met. In this way, the function of the arbitrary door for text and pictures can be realized.

[0337] In the foregoing scenarios one to five, the conflict condition is a condition that can respond to a continuous long pressing operation or a long pressing and dragging operation.

[0338] For example, the condition that can respond to a continuous long pressing operation includes: in a video scenario, continuously long pressing a playing video to realize continuous fast forwarding; in a recording scenario, continuously long pressing a recording control to realize continuous recording; in a game scenario, continuously long pressing a game screen to realize continuous triggering of skills (continuous acceleration, continuous shooting, etc.).

[0339] For example, the condition that can respond to a long pressing and dragging operation includes: in a scenario where a control itself is draggable, long pressing and dragging the control to realize deletion or adjustment of order; in other scenarios, long pressing and dragging a preset constituent element to realize adjustment of order.

[0340] In these conflict conditions, by conflict identification, the arbitrary door is only responded to in the case of no conflict, thereby accurately excluding the conflict conditions.

[0341] In some scenarios that can respond to one-off long press, there can also be conflicts. Referring to FIG. 19, the phone can display an interface 1901, which includes a picture 19011 on a social media. Generally, in response to a user's long press operation on the picture 19011, the phone can display an interface 1902, which can provide some general services for the picture 19011, i.e., the same services that can be provided for each picture, such as services for sharing to each platform A, B, C, D, etc., a service for saving the picture, a service for viewing the original picture, etc. At the same time, as described above, long press and drag can also trigger an arbitrary gate. Therefore, for the user's long press operation on the picture 19011, the phone cannot determine whether to provide general services or further respond to the drag of the arbitrary gate, and thus conflicts can occur.

[0342] Based on this, in some embodiments, the long press duration 1 for responding to the drag of the arbitrary gate can be different from the long press duration 2 for responding to the one-off long press operation, and the responses are based on different long press durations, so as to avoid conflicts. The long press duration 1 is greater than the long press duration 2.

[0343] In a specific implementation (referred to as way one), after detecting that the long press operation reaches the long press duration 2, the phone can respond to the one-off long press operation. Then, after detecting that the long press operation reaches the long press duration 1, the phone can respond to the drag operation of the arbitrary gate. It should be noted that if the long press operation ends before reaching the long press duration 1, the drag operation of the arbitrary gate will not be responded to. In this way, different responses can be performed at different time points, and conflicts can be avoided.

[0344] Referring back to FIG. 19, the user performs a long press operation on the picture 19011, and after the long press operation reaches the long press duration 2, the phone can display the interface 1902, i.e., responding to the one-off long press operation. Then, the user still performs a long press operation on the picture 19011, and after the long press operation reaches the long press duration 1, in response to the user's drag operation on the picture 19011, the phone can display an interface 1903, which includes shortcut entries of various services matched with the picture 19011, i.e., responding to the arbitrary gate.

[0345] In another specific implementation (referred to as way two), in the case where the long press operation reaches the long press duration 2 but ends before reaching the long press duration 1, the phone can respond to the one-off long press operation. In the case where the long press operation reaches the long press duration 2 and has not ended, the phone can respond to the drag operation of the arbitrary gate. In this way, one response can be performed for different situations of the long press duration, and conflicts can also be avoided.

[0346] In combination with the above solution for resolving the conflict of the one-time long press operation, in the flow of FIG. 17, after detecting that the conflict condition is not met in S1701, and if the scenario belongs to the one-time long press, S1703 can be replaced with: the mobile phone responds to the one-time long press operation and the drag operation of the arbitrary door based on the long press duration 1 and the long press duration 2, specifically as shown in the above manner one or manner two. Thus, the conflict of the one-time long press operation and the response of the arbitrary door can be further avoided in the case of excluding the conflict of scenarios one to five.

[0347] It should be noted that the mobile phone (such as the corresponding application) can determine whether the scenario belongs to the one-time long press based on the business logic, so as to determine whether to respond to the one-time long press.

[0348] The embodiment of the present application also provides an electronic device, which can include a display screen, a memory and one or more processors (such as CPU, GPU, NPU, etc.). The display screen, the memory and the processor are coupled. The memory is used to store computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device can perform various functions or steps performed by the device in the above method embodiments.

[0349] The embodiment of the present application also provides a chip system, which includes at least one processor and at least one interface circuit. The processor and the interface circuit can be interconnected through a circuit. For example, the interface circuit can be used to receive signals from other devices (such as the memory of the electronic device). For another example, the interface circuit can be used to send signals to other devices (such as the processor). Illustratively, the interface circuit can read the instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which are not limited in the embodiment of the present application.

[0350] The embodiment of the present application also provides a computer storage medium, which stores computer instructions. When the computer instructions are run on the electronic device, the electronic device performs the above related method steps to implement the image processing method in the above embodiments.

[0351] The embodiment of the present application also provides a computer program product, which makes the computer execute the above related steps to implement the image processing method in the above embodiments when the computer program product is run on the computer.

[0352] In addition, the embodiment of the present application further provides a device, which can be a chip, a component or a module, and the device can include a processor and a memory connected to each other; the memory is used to store computer-executed instructions; when the device is running, the processor can execute the computer-executed instructions stored in the memory, so that the chip executes the image processing method in each method embodiment described above.

[0353] The electronic device, the computer storage medium, the computer program product or the chip provided in the embodiment are used to execute the corresponding method provided above, and thus the beneficial effects achieved by the electronic device, the computer storage medium, the computer program product or the chip can refer to the beneficial effects of the corresponding method provided above, which will not be described here again.

[0354] Through the description of the above embodiments, those skilled in the art can clearly understand that, for the convenience and brevity, only the division of the above functional modules is taken as an example, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0355] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0356] The units described as separate components can or can not be physically separate, and the components shown as units can be one physical unit or multiple physical units, that is, can be located in one place or distributed in multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0357] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0358] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application essentially or in other words the parts that make contributions to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The software product is stored in a storage medium and includes a plurality of instructions for causing an apparatus (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0359] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit them. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the present application.

Claims

An interaction conflict resolution method characterized by The method is applied to an electronic device, and the method comprises: displaying a first interface in which a first video is being played; in response to a long press operation on the first video, controlling playing of the first video; displaying a second interface in which a second video is paused; in response to a long press and drag operation on the second video, displaying shortcut entries of a plurality of first services while moving a thumbnail of the second video along with the drag. The method of claim 1, wherein The controlling of the playing of the first video comprises fast forwarding the first video. The method according to claim 1 or 2, characterized in that The first interface and the second interface are not application interfaces of a gallery application. The method according to any one of claims 1-3, characterized in that The first video and the second video are the same video. After the controlling of the playing of the first video in response to the long press operation on the first video, the displaying of the second interface comprises: in response to a pause playing operation on the first video, displaying the second interface. The method according to any one of claims 1-3, characterized in that The displaying of the first interface and the displaying of the second interface comprise: simultaneously displaying the first interface and the second interface in the foreground. The method according to any one of claims 1-5, characterized in that The controlling of the playing of the first video in response to the long press operation on the first video comprises: in response to the long press operation on the first video, detecting that the pressed video is a video being played, and controlling the playing of the first video. The method according to any one of claims 1-6, characterized in that The displaying of the shortcut entries of the plurality of first candidate services while moving the thumbnail of the second video along with the drag in response to the long press and drag operation on the second video comprises: in response to the long press and drag operation on the second video, detecting that the pressed video is not a video being played, and displaying shortcut entries of a plurality of first services while moving the thumbnail of the second video along with the drag. The method according to any one of claims 1-7, characterized in that The first interface further comprises first content, and the first content is not a video. After the displaying of the first interface, the method further comprises: in response to a long press and drag operation on the first content, detecting that the pressed content is not a video being played, and displaying shortcut entries of a plurality of second services while moving a thumbnail of the first content along with the drag. The method according to any one of claims 1-8, characterized in that The pressed video is detected to be a video being played when at least one of the following conditions is met: the video being played; the pressed object is a video playing window; and the application being pressed and the application in which the video is being played are the same application. The method according to any one of claims 1-9, characterized in that After the displaying of the shortcut entries of the plurality of first services, the method further comprises: displaying a third interface, the third interface comprising second content; receiving a long press operation of the second content by a user; in a case where it is detected that a conflict condition is met, performing a long press response or a drag response of the second content; in a case where it is detected that the conflict condition is not met, displaying shortcut entries of a plurality of third services while moving a thumbnail of the second content along with a drag in response to a drag operation on the second content. The method of claim 10, wherein The conflict condition comprises at least one of the following: the target application is an application that is calling a microphone to collect sound and / or a camera to collect images, and the target application is an application corresponding to the third interface; the second content supports a drag response; the third interface comprises a game screen, and the second content is content in the game screen; and ​ The second content is a preset constituent element in the third interface. The method according to claim 10 or 11, characterized in that The method further includes: In response to the long-press operation on the second content reaching a first time length, in response to a drag operation on the second content, a thumbnail of the second content is moved along with the drag operation, and shortcut entrances of a plurality of third services are displayed. Before the long-press operation on the second content reaches the first time length, the method further includes: After the long-press operation on the second content reaches a second time length, a long-press menu bar of a target application is displayed, the target application being an application corresponding to the third interface. The method according to claim 10 or 11, characterized in that The method further includes: In response to the long-press operation on the second content reaching a first time length, in response to a drag operation on the second content, a thumbnail of the second content is moved along with the drag operation, and shortcut entrances of a plurality of third services are displayed. The method further includes: After the long-press operation on the second content reaches a second time length but does not reach the first time length, a long-press menu bar of a target application is displayed, the target application being an application corresponding to the third interface, the second time length being less than the first time length. An electronic device, characterized by comprising: The method further includes: A display screen, one or more processors, and one or more memories; the one or more processors being coupled with the display screen and the one or more memories; the one or more memories being used to store computer program codes, the computer program codes including computer instructions, when the one or more processors execute the computer instructions, causing the electronic device to perform the method as claimed in any one of claims 1-13. A computer readable storage medium having stored thereon computer instructions, characterized in that, When the computer instructions run on an electronic device, the electronic device performs the method as claimed in any one of claims 1-13. A computer program product comprising computer instructions, characterized in that, When the computer program product runs on a computer, the computer performs the method as claimed in any one of claims 1-13.

Images