Information circulation method, electronic equipment, readable storage medium and chip system

Abstract

The invention discloses an information circulation method, electronic equipment, a readable storage medium and a chip system.The method can comprise the steps that the electronic equipment displays a browsing interface of a first application, and after circulation triggering operation of a user is detected, the currently-displayed browsing interface of the first application is automatically screenshot, and a screenshot image is obtained; then, according to the position of the circulation triggering operation in the display screen, identifying to-be-circulated information expected to be circulated by the user from the screenshot image; at least one application matched with the intention type of the to-be-transferred information is obtained through prediction, and an icon of the at least one application is displayed; and finally, based on the stop position of the dragging operation of the user, determining a second application for information circulation, and controlling the to-be-circulated information to be circulated to the second application. According to the invention, the flexibility and intelligence of information circulation can be improved.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an information transfer method, electronic device, readable storage medium and chip system. Background Technology

[0002] With the continuous development of terminal technology, the applications installed on electronic devices are becoming increasingly diverse. Among these, different applications support the flow of information (such as images and text), meaning information can be transferred from one application to another. When transferring information between applications, the electronic device can first receive a user's save command for the information to be transferred within one application's interface, then save the information locally, and then launch the other application. Within that application's interface, it will receive a user's selection command for the locally saved information to be transferred, thus achieving the flow of information from one application to another. However, this method of information transfer is cumbersome and lacks a high level of intelligence. Summary of the Invention

[0003] This application provides an information transfer method, an electronic device, a readable storage medium, and a chip system, which are beneficial for improving the flexibility and intelligence of information transfer between applications.

[0004] In a first aspect, this application provides an information flow method, the method comprising: displaying a first user interface, the first user interface including a browsing interface of a first application; in response to detecting a flow trigger operation acting on the browsing interface, acquiring a screenshot image corresponding to the browsing interface; determining information to be flowed from the screenshot image based on the location information of the flow trigger operation, and outputting a drag shadow of the information to be flowed on the browsing interface; in response to detecting a drag operation acting on the drag shadow, determining at least one candidate application matching the intent type corresponding to the information to be flowed; outputting a flow option bar on the first user interface, the flow option bar including icons of at least one candidate application; determining a second application from the at least one candidate application based on the stopping position of the drag operation in the flow option bar; and controlling the information to be flowed to flow to the second application.

[0005] In the above embodiments, after detecting a user's flow trigger operation, the electronic device can automatically take a screenshot of the browsing interface of the currently displayed first application, identify the information to be flowed in the browsing interface based on the location information of the flow trigger operation, and then flow the information to be flowed to other applications according to the user's drag operation. In this way, the automation and intelligence of information flow can be realized, which is conducive to improving the efficiency of information flow between applications.

[0006] In conjunction with the first aspect, in one possible approach, determining the information to be transferred from the screenshot image based on the location information of the transfer trigger operation includes: identifying the location information of candidate information in the screenshot image; and determining the information to be transferred from the candidate information based on the location information of the candidate information in the screenshot image and the location information of the transfer trigger operation.

[0007] In this technical solution, the electronic device can automatically identify text and / or images in a screenshot, and, by combining the location of the user's action on the screen to trigger the flow, as well as the location of the text and / or images in the screenshot, determine the information the user wishes to transfer. By intelligently identifying the information the user wishes to transfer, subsequent information transfer can be achieved more efficiently.

[0008] In conjunction with the first aspect, in one possible approach, the candidate information includes at least one image, and the position information of an image includes the vertex position of the bounding rectangle corresponding to the image; the above-mentioned determination of the information to be transferred from the candidate information based on the position information of the candidate information in the screenshot image and the position information of the transfer trigger operation includes: determining that the transfer trigger operation is within the first bounding rectangle based on the vertex position of the bounding rectangle corresponding to at least one image and the position information of the transfer trigger operation; and determining the image corresponding to the first bounding rectangle as the information to be transferred.

[0009] In this technical solution, if the electronic device determines that the location triggering the transfer operation is within the bounding rectangle corresponding to the image, then it can assume that the information the user expects to transfer is the image. In this way, the information the user expects to transfer can be accurately identified.

[0010] In conjunction with the first aspect, in one possible approach, the above method further includes: generating a drag shadow of the information to be transferred based on the image corresponding to the first circumscribed rectangle.

[0011] In this technical solution, if the information to be transferred is an image, a drag shadow can be generated based on the bounding rectangle corresponding to the image and the content of the image, so as to more intuitively reflect the drag position of the drag operation on the display screen.

[0012] In conjunction with the first aspect, in one possible approach, the candidate information includes at least one piece of text, and the position information of the text includes the vertex positions of the corresponding bounding polygon of the text; the above-mentioned determination of the information to be transferred from the candidate information based on the position information of the candidate information in the screenshot image and the position information of the transfer trigger operation includes: determining that the transfer trigger operation is within the first bounding polygon based on the vertex positions of the bounding polygon corresponding to at least one piece of text and the position information of the transfer trigger operation; and determining the text corresponding to the first bounding polygon as the information to be transferred.

[0013] In this technical solution, if the electronic device determines that the location triggering the flow operation is within the circumscribed polygon corresponding to the text, then it can consider the information the user expects to flow as text. In this way, the information the user expects to flow can be accurately identified.

[0014] In conjunction with the first aspect, in one possible approach, the candidate information includes at least one image and at least one piece of text. The positional information of an image includes the vertex positions of the bounding rectangle corresponding to the image, and the positional information of a piece of text includes the vertex positions of the bounding polygon corresponding to the text. The above-mentioned determination of the information to be transferred from the candidate information based on the positional information of the candidate information in the screenshot image and the positional information of the transfer trigger operation includes: determining that the transfer trigger operation is within the second bounding rectangle and within the second bounding polygon based on the vertex positions of the bounding rectangle corresponding to at least one image, the vertex positions of the bounding polygon corresponding to at least one piece of text, and the positional information of the transfer trigger operation; and determining the text corresponding to the second bounding polygon as the information to be transferred.

[0015] In this technical solution, if the electronic device determines that the location triggering the flow operation is within both the bounding rectangle of the image and the bounding polygon of the text, then it can be assumed that the information the user expects to flow is text. In this case, it can be understood that the text may be the text displayed on the image, and the electronic device is more likely to determine that the user has selected text.

[0016] In conjunction with the first aspect, in one possible approach, the above method further includes: inputting the information to be transferred into an intent recognition model to obtain the intent type corresponding to the information to be transferred output by the intent recognition model.

[0017] In this technical solution, the intent recognition model can be a single, unified model or a combination of multiple models. Electronic devices utilize this intent recognition model to identify the intent type corresponding to the information to be transferred, thereby improving the intelligence of information flow.

[0018] In conjunction with the first aspect, in one possible approach, the icons of at least one candidate application are sorted in descending order of application priority.

[0019] In this technical solution, applications with higher priority are more likely to be selected and used by users. By intelligently arranging the order of applications, the intelligence of human-computer interaction can be improved.

[0020] In conjunction with the first aspect, in one possible approach, the flow trigger operation is a press operation with a duration greater than or equal to a time threshold.

[0021] In conjunction with the first aspect, in one possible manner, the above method further includes: jumping from the first user interface to the user interface of the second application, where the user interface of the second application displays the information to be transferred.

[0022] In this technical solution, after the electronic device controls the flow of information to be transferred to the second application, it can display the user interface of the second application to more intuitively demonstrate the effect of information transfer.

[0023] Secondly, this application provides an electronic device comprising: one or more processors, a display screen, and a memory; the memory is coupled to the one or more processors and is used to store computer program code, the computer program code including computer instructions, the one or more processors calling the computer instructions to cause the electronic device to perform: displaying a first user interface, the first user interface including a browsing interface of a first application; in response to detecting a flow trigger operation acting on the browsing interface, acquiring a screenshot image corresponding to the browsing interface; determining information to be flowed from the screenshot image based on the position information of the flow trigger operation, and outputting a drag shadow of the information to be flowed on the browsing interface; in response to detecting a drag operation acting on the drag shadow, determining at least one candidate application matching the intent type corresponding to the information to be flowed; outputting a flow option bar on the first user interface, the flow option bar including icons of at least one candidate application; determining a second application from the at least one candidate application based on the stop position of the drag operation in the flow option bar; and controlling the flow of the information to be flowed to the second application.

[0024] In conjunction with the second aspect, in one possible manner, the one or more processors invoke the computer instructions to cause the electronic device to perform: identifying the location information of candidate information in a screenshot image; and determining the information to be transferred from the candidate information based on the location information of the candidate information in the screenshot image and the location information of the transfer trigger operation.

[0025] In conjunction with the second aspect, in one possible approach, the candidate information includes at least one image, and the location information of an image includes the vertex position of the bounding rectangle corresponding to the image; the one or more processors invoke the computer instructions to cause the electronic device to perform: determining that the flow triggering operation is within a first bounding rectangle based on the vertex position of the bounding rectangle corresponding to at least one image and the location information of the flow triggering operation; and determining the image corresponding to the first bounding rectangle as the information to be flowed.

[0026] In conjunction with the second aspect, in one possible manner, the one or more processors invoke the computer instructions to cause the electronic device to perform: generating a drag shadow of the information to be transferred based on the image corresponding to the first circumscribed rectangle.

[0027] In conjunction with the second aspect, in one possible manner, the candidate information includes at least one piece of text, and the location information of the text includes the vertex positions of the corresponding circumscribed polygon of the text; the one or more processors invoke the computer instructions to cause the electronic device to perform: determining the flow triggering operation within the first circumscribed polygon based on the vertex positions of the circumscribed polygon corresponding to at least one piece of text and the location information of the flow triggering operation; and determining the text corresponding to the first circumscribed polygon as the information to be flowed.

[0028] In conjunction with the second aspect, in one possible approach, the candidate information includes at least one image and at least one piece of text. The location information of the image includes the vertex positions of the bounding rectangle corresponding to the image, and the location information of the text includes the vertex positions of the bounding polygon corresponding to the text. The one or more processors invoke the computer instructions to cause the electronic device to perform: determining, based on the vertex positions of the bounding rectangle corresponding to at least one image, the vertex positions of the bounding polygon corresponding to at least one piece of text, and the location information of the flow triggering operation, that the flow triggering operation is within the second bounding rectangle and within the second bounding polygon; and determining the text corresponding to the second bounding polygon as the information to be flowed.

[0029] In conjunction with the second aspect, in one possible manner, the one or more processors invoke the computer instructions to cause the electronic device to perform: inputting the information to be transferred into the intent recognition model, and obtaining the intent type corresponding to the information to be transferred output by the intent recognition model.

[0030] In conjunction with the second aspect, in one possible approach, the icons of at least one candidate application are sorted in descending order of application priority.

[0031] In conjunction with the second aspect, in one possible approach, the flow trigger operation is a press operation with a duration greater than or equal to a time threshold.

[0032] In conjunction with the second aspect, in one possible manner, the one or more processors invoke computer instructions to cause the electronic device to perform: a jump from a first user interface to the user interface of a second application, the user interface of the second application displaying information to be transferred.

[0033] Thirdly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0034] Fourthly, this application provides a chip system coupled to a memory, the chip system being used to read and execute a computer program stored in the memory to implement the method described in the first aspect above.

[0035] Fifthly, this application provides a computer program product containing instructions that, when run on an electronic device, cause the electronic device to perform the method described in the first aspect. Attached Figure Description

[0036] Figure 1 A schematic diagram of an information flow provided in an embodiment of this application;

[0037] Figure 2 This is a schematic diagram of another information flow provided in an embodiment of this application;

[0038] Figure 3 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0039] Figure 4 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0040] Figure 5 This application provides a schematic diagram of inter-module interaction in the software structure of an electronic device.

[0041] Figure 6 A flowchart illustrating an information transfer method provided in an embodiment of this application;

[0042] Figures 7A-7D A schematic diagram of an information flow method provided in an embodiment of this application;

[0043] Figures 8A-8D A schematic diagram of an interface for another information flow method provided in an embodiment of this application;

[0044] Figure 9 This is a flowchart illustrating another information flow method provided in an embodiment of this application. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0046] The terms "first," "second," "third," etc., used in the embodiments of this application are to distinguish different objects, rather than to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, it may include a series of steps or units, or optionally, steps or units not listed, or other steps or units inherent to these processes, methods, products, or devices. The terms "one embodiment" or "some embodiments," etc., mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of the embodiments of this application, do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0047] Furthermore, "at least one" refers to one or more, while "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can mean: a, or b, or c, or a and b, or a and c, or b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0048] The term "user interface (UI)" used in the following embodiments of this application refers to the medium interface through which an application or operating system interacts and exchanges information with the user. It realizes the conversion between the internal form of information and the form that the user can accept. The user interface is source code written in a specific computer language such as Java or Extensible Markup Language (XML). The interface source code is parsed and rendered on the electronic device, ultimately presenting content that the user can recognize. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be visible interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets displayed on the screen of an electronic device.

[0049] As applications become increasingly diverse, electronic devices often have multiple applications installed, and information (such as images and text) can flow between different applications. For example, information from application A can flow to application B. See [example example]. Figure 1 , Figure 1 This is a schematic diagram illustrating an information flow process provided in an embodiment of this application. For example... Figure 1 As shown in (a), the electronic device can display the chat interface of a chat application, which includes a payment QR code image. Figure 1 As shown in (b), in response to detecting a save operation for the payment QR code image, the electronic device can save the payment QR code icon locally. Upon receiving a user's command to return to the home screen, the electronic device can return to the home screen, which includes the payment application. In response to detecting a selection operation for the payment application, the electronic device displays the payment application's user interface. Figure 1 As shown in (c), the user interface of the payment application includes a QR code scanning control. In response to detecting a selection operation on the QR code scanning control, the electronic device can display a QR code scanning interface. Figure 1 As shown in (d), the scanning interface includes a photo album control. In response to a detected selection operation on the photo album control, the electronic device can display at least one locally saved image, including a payment QR code image saved from the chat interface. Thus, in response to receiving a selection operation on the payment QR code image, the electronic device can control the payment application to scan the payment QR code image, enabling the transfer of the payment QR code image from the chat application to the payment application.

[0050] for Figure 1 The information flow method shown involves multiple human-computer interactions and requires frequent application switching, such as application A -> desktop -> application B, which reduces the efficiency of electronic devices in processing data. To improve this interactive experience, an information flow scheme is proposed that allows electronic devices to directly and quickly transfer information from one application to another without having to return to the desktop from one application's interface and then launch another application.

[0051] For example, please see Figure 2 , Figure 2 This is a schematic diagram illustrating another information flow process provided in an embodiment of this application. For example... Figure 2 As shown in (a), the electronic device can display a browsing interface for a news application, which includes text and images. In response to detecting a selection operation on text 201, the electronic device can determine whether the selected text 201 is a draggable element. Figure 2As shown in (b), when text 201 is a draggable element, the electronic device can, for example, obtain the text content based on Android's TextView (a UI component for managing text resources) control mechanism and display the drag shadow corresponding to text 201. Figure 2 As shown in (c), in response to detecting a drag operation on the drag shadow corresponding to text 201, an icon of at least one application can be displayed, such as a favorites application, a search application, a sharing application, a social application, etc. Figure 2 As shown in (d), in response to detecting that the drag shadow corresponding to text 201 has been dragged onto the icon of the favorites application, the electronic device can save text 201 to the favorites application and display a prompt message 202 to indicate successful saving. If the electronic device determines that text 201 is a non-draggable element, the user needs to manually trigger the electronic device's screenshot browsing interface, for example, by pressing a button or using a gesture to take a screenshot. After acquiring the screenshot image, the electronic device identifies the text and / or image in the screenshot image and displays the identified text and / or image. Furthermore, the user can manually select the text and / or image they wish to transfer.

[0052] Figure 2 The information flow method shown is compared to Figure 1 The information flow method shown reduces the application switching process. However, Figure 2 The information flow method shown requires users to manually select the desired information, which is limited by the Android-based TextView and ImageView (UI components for managing image resources) controls. Furthermore, users are unaware whether the selected information is a draggable element on the screen. For non-draggable elements, a screenshot must be manually triggered, and the desired information must then be selected from valid elements (such as text and / or images) recognized by the electronic device. This information flow process is cumbersome and lacks flexibility and intelligence.

[0053] Based on this, embodiments of this application provide an information flow method that can be applied to electronic devices. The electronic device can automatically identify the information the user wishes to flow based on the location of the user-triggered information flow operation on the interface, without requiring the user to manually select information or perform manual screenshots, thus providing a more convenient information flow method. This approach enables more efficient information flow and improves the flexibility and intelligence of information flow.

[0054] In the embodiments of this application, the electronic device may be a mobile phone, tablet computer, laptop computer, handheld computer, mobile internet device (MID), wearable device, virtual reality (VR) device, augmented reality (AR) device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, personal digital assistant (PDA), etc., and the embodiments of this application are not limited thereto.

[0055] To facilitate understanding, the structure of the electronic device in the embodiments of this application will be described below.

[0056] For example, please see Figure 3 , Figure 3 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0057] like Figure 3 As shown, the electronic device may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0058] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0059] An NPU (Neural Processing Unit) is a neural network computing processor that, by drawing inspiration from the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, rapidly processes input information and can continuously learn on its own. NPUs can enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding. In this embodiment, the NPU can be used to recognize screenshot images and predict the intent type corresponding to images and / or text.

[0060] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0061] Electronic devices implement display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0062] The wireless communication function of electronic devices can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0063] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0064] The mobile communication module 150 can provide solutions for wireless communication applications including 2G / 3G / 4G / 5G in electronic devices. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0065] The wireless communication module 160 can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLAN) (such as WiFi networks), Bluetooth (BT), BLE broadcasting, global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0066] Electronic devices implement display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0067] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include one or N display screens 194, where N is a positive integer greater than 1. In this embodiment, the display screen 194 can be used to display images, videos, etc. Figures 7A-7D ,as well as Figures 8A-8D The interface shown.

[0068] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch screen." Touch sensor 180K detects touch operations applied to or near it. Touch sensor 180K can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of the electronic device, in a different position than display screen 194. In this embodiment, touch sensor 180K can be used to detect user-triggered information flow operations, such as long press operations.

[0069] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0070] The hardware structure of electronic devices has been introduced above. The software structure of electronic devices will be introduced below.

[0071] In this embodiment, the software architecture of the electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment uses the layered architecture Android system as an example to illustrate the software architecture of the electronic device.

[0072] For example, please see Figure 4 , Figure 4 This is a schematic diagram of the software structure of an electronic device provided in an embodiment of this application. For example... Figure 4 As shown, the layered architecture of an electronic device divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system consists of, from top to bottom, the application layer, the application framework layer, and the kernel layer, etc.

[0073] The application layer can include a series of application packages, which may include applications such as camera, gallery, chat, map, calendar, music, call, favorites, and video.

[0074] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0075] In this embodiment of the application application, the application framework layer may include a business access layer, a business framework layer, and a basic framework layer.

[0076] The business access layer can be understood as the interface adaptation layer of the business framework layer, and may include intelligent workflow modules, visual access modules, intent access modules, service access modules, and intelligent multi-window modules. For example, the intelligent workflow module can call the capabilities of the business framework layer through APIs, and then adapt and transform them to provide the corresponding capabilities to the application layer.

[0077] The business framework layer may include a content recognition module, a content recognition server, a natural language understanding (NLU) platform, a visual platform, an intent platform, and a service management platform.

[0078] The content recognition module can be used to initiate requests to recognize image content, and the content recognition server can be used to respond to requests initiated by the content recognition module and interact with the visual platform.

[0079] The NLU platform's functions can include address recognition, point of interest (POI) recognition, video / movie recognition, and time recognition. It is primarily responsible for identifying the entity types of text. For example, it can call an NLU model to identify entity types, including but not limited to addresses, videos / movies, music, time, and names.

[0080] The functions of a visual platform can include layout analysis, code recognition, tag classification, and text detection. It is primarily responsible for identifying text and / or image regions in screenshots, for example, by using computer vision (CV) models.

[0081] The intent platform's functions include intent recognition, intent prediction, intent querying, and service ranking. It is primarily responsible for predicting the intent type of text and / or images, querying applications that match the intent type, and ranking the queried applications to obtain an ordered application list.

[0082] The service management platform's functions include service registration, service updates, service matching, and client-cloud synchronization. It is primarily responsible for the integration of intelligent workflow services.

[0083] In some embodiments, the capabilities of each module in the application framework layer of an electronic device may rely on the capabilities of a cloud-side server. This cloud-side server may be configured with a smart cloud platform. The smart cloud platform can use large amounts of data for model training and supports the deployment of the trained model on the electronic device.

[0084] The basic framework layer mainly refers to the Android basic framework upon which device human-computer interaction depends. This can include the View framework, drag-and-drop framework, Activity Manager Service (AMS), and Window Manager Service (WMS). The View framework can dispatch and process operation events according to the current layout of the electronic device's interface, passing the events to the corresponding application controls. The drag-and-drop framework can be used to detect user drag-and-drop operations. AMS provides Android with system services for managing the running state of Activities, as well as managing the running state of other components in Android. WMS is used to manage window applications, and can obtain the size of the electronic device's screen, determine whether there is a status bar, lock the electronic device's screen, capture the screen, create precise selection boxes, and create precise display areas, etc.

[0085] It should be understood that the application framework layer may also include Figure 4 The following are not shown: Resource Manager, Notification Manager, Input Manager, etc. Resource Manager provides applications with various resources, such as localized strings, icons, images, layout files, video files, etc. Notification Manager allows applications to display notifications in the status bar; it can be used to convey informative messages and can disappear automatically after a short pause without user interaction. Input Manager manages all input methods across the system, including the keyboard, touchscreen, etc.

[0086] The kernel layer is the layer between hardware and software. The kernel layer includes at least display drivers, camera drivers, audio drivers, sensor drivers, and processor drivers.

[0087] It should be noted that the functional modules included in the above software structure are merely exemplary and do not constitute a specific limitation on the mobile phone software architecture of this application. In other embodiments, the functional modules included in the above software structure may be more or fewer, and this application does not impose any limitations on this. Although the embodiments of this application use the Android system as an example for illustration, its basic principles are equally applicable to electronic devices based on operating systems such as iOS or Windows.

[0088] To facilitate understanding, the following is a schematic description of the interaction process between modules in the software structure of electronic devices.

[0089] For example, please see Figure 5 , Figure 5 This is a schematic diagram illustrating the interaction between modules in the software structure of an electronic device, as provided in an embodiment of this application. Figure 5 As shown, ① After AMS, View framework, and WMS detect the user's information flow trigger operation, they can report the information flow trigger event to the content recognition module. ② The content recognition module obtains a screenshot of the current interface and the location information of the user's trigger operation, and passes the screenshot and the location information of the trigger operation to the content recognition module. ③ The content recognition module can send the screenshot to the visual middle platform. ④ The visual middle platform recognizes the text and / or images in the screenshot and provides the recognition results to the content recognition server. ⑤ The content recognition server can combine the location information of the trigger operation and the recognition results of the screenshot to determine the information to be flowed by the user, and return the determined information to be flowed to the content recognition module, so that the content recognition module can output the drag shadow of the information to be flowed. ⑥ When the drag framework detects the user's drag operation on the drag shadow, it can report the drag event to the content recognition module. ⑦ The content recognition module calls the drag framework's capabilities to launch the intelligent flow service. ⑧ The drag framework calls the intelligent flow module to realize the flow of the information to be flowed.

[0090] The overall process of the information flow method provided in the embodiments of this application will be described in detail below.

[0091] Please see Figure 6 , Figure 6 This is a flowchart illustrating an information flow method provided in an embodiment of this application. Figure 6 As shown, this method can be applied to electronic devices, and may include, but is not limited to, the following steps:

[0092] S601 displays the first user interface.

[0093] In this embodiment, the first user interface includes a browsing interface of a first application. The browsing interface of the first application can be understood as an interface used to display information within the first application, including but not limited to text, images, videos, audio, or other media types. For example, using... Figure 2 Taking the scenario shown as an example, Figure 2 The interface shown in (a) above, which displays news content, can be understood as the browsing interface in a news application. Similarly, in a shopping scenario, the product list interface, product details interface, and shopping cart interface in a shopping application can all be understood as browsing interfaces within the shopping application.

[0094] S602, in response to detecting a flow trigger operation acting on the browsing interface, acquires a screenshot image corresponding to the browsing interface.

[0095] In this embodiment, the flow triggering operation can be understood as the operation that triggers the information flow function of the electronic device. It should be noted that the information flow in this embodiment can also be replaced by business flow, data flow, etc., and this application does not limit it.

[0096] Optionally, the flow trigger operation can be a press operation with a duration greater than or equal to a time threshold, or described as a long press operation. For example, the time threshold can be a preset time of 3 seconds. If the electronic device detects that the user's finger presses the browsing interface of the first application for a duration greater than or equal to 3 seconds (e.g., 4 seconds), it can trigger the information flow function. The electronic device can, for example, detect the flow trigger operation of the user on the interface of the first application through a touch sensor. It should be noted that this embodiment uses a long press operation as an example for flow trigger operation. In other embodiments, the flow trigger operation can also be other trigger operations, such as double-tapping the screen, single-tapping the screen, knuckle selection, or other preset operations.

[0097] After the electronic device detects a flow trigger operation on the browsing interface of the first application, it can determine that the user expects to extract information from the current interface. Then, it can automatically capture the browsing interface of the first application and obtain the corresponding screenshot image of the browsing interface, so as to facilitate subsequent identification of the information contained in the screenshot image of the browsing interface.

[0098] S603 determines the information to be transferred from the screenshot image based on the location information of the transfer trigger operation, and outputs the drag shadow of the information to be transferred on the browsing interface.

[0099] In this embodiment, the location information of the flow trigger operation can be the specific location where the user's finger touches the screen of the electronic device. For example, if the flow trigger operation is a long press, the location information can be the position where the user's finger is placed when long-pressing on the screen; this position can also be described as the touch point position of the long press operation. If the flow trigger operation is a knuckle selection operation, the location information can be the trajectory position selected by the user on the screen. Based on the location information of the flow trigger operation, the electronic device can determine the information to be flowed that the user expects to flow from the screenshot image. Then, a drag shadow of the information to be flowed can be output on the browsing interface of the first application. Here, the drag shadow of the information to be flowed can be a semi-transparent image or outline, and this drag shadow can move with the user's drag operation so that the user can know the object being dragged and the drag position.

[0100] In some embodiments, the electronic device determines the information to be transferred from a screenshot image based on the location information of the transfer trigger operation by first identifying the content of the screenshot image to obtain candidate information; then, based on the location information of the candidate information and the location information of the transfer trigger operation, determining the information to be transferred from the candidate information. Here, candidate information can be understood as information in the screenshot image that the electronic device recognizes as supporting transfer, and information to be transferred can be understood as information that the electronic device recognizes as the user's desired transfer. Optionally, the candidate information in the screenshot image may include text, images, etc.

[0101] In some embodiments, candidate information in a screenshot image may include at least one image, and the position information of an image may include the vertex position of the bounding rectangle corresponding to that image. Based on the vertex position of the bounding rectangle corresponding to at least one image and the position information of the flow trigger operation, the electronic device can determine which bounding rectangle corresponding to the flow trigger operation is applied to. Furthermore, the electronic device can determine the image corresponding to the bounding rectangle where the flow trigger operation is located as the information to be flowed that the user expects to flow. For example, assuming the screenshot image includes images A, B, and C, if the electronic device determines that the flow trigger operation is applied to the bounding rectangle corresponding to image C based on the vertex positions of the bounding rectangles corresponding to images A, B, and C, then the electronic device can determine that the information to be flowed is image C.

[0102] Optionally, before determining the information to be transferred from at least one image, the electronic device can filter out images whose bounding rectangle area is less than or equal to an area threshold. This is because images smaller than a certain size in screenshots may be images without actual content, such as decorative icons used to enhance the interface's aesthetics. Filtering out these images before identifying the information to be transferred improves the accuracy of the identification process.

[0103] In some embodiments, the candidate information in the screenshot image may include at least one piece of text, and the location information of the text may include the vertex positions of the bounding polygon corresponding to the text. Optionally, a piece of text may be divided according to dimensions such as blocks, paragraphs, or lines, which is not limited in this application. Similarly, based on the vertex positions of the bounding polygon corresponding to at least one piece of text and the location information of the flow trigger operation, the electronic device can determine which bounding polygon corresponding to the text the flow trigger operation is specifically applied to. Furthermore, the electronic device can determine the text corresponding to the bounding polygon where the flow trigger operation is located as the information to be flowed that the user expects to flow.

[0104] In some embodiments, candidate information in a screenshot image may include at least one image and at least one piece of text. That is, the electronic device recognizes that the content in a screenshot image may include both images and text. In this case, based on the vertex positions of the bounding rectangle corresponding to at least one image, the vertex positions of the bounding polygon corresponding to at least one piece of text, and the position information of the flow trigger operation, the electronic device determines the information to be flowed as the image corresponding to the first bounding rectangle if it is determined that the flow trigger operation is located within the first bounding rectangle; if it is determined that the flow trigger operation is located within the first bounding polygon, the information to be flowed can be determined as the text corresponding to the first bounding polygon; if it is determined that the flow trigger operation is located within both the second bounding rectangle and the second bounding polygon, the information to be flowed can be preferentially determined as text; if it is determined that the flow trigger operation is neither within a bounding rectangle nor within a bounding polygon, the flow trigger operation can be considered invalid, and subsequent information flow processes will not be triggered.

[0105] Optionally, if an electronic device detects that the flow trigger operation is located within both the bounding rectangle of an image and the bounding polygon of a piece of text, it can be understood that the position of the text on the interface coincides with the position of the image on the interface; that is, the text can be the text displayed on the image. In this case, the electronic device can determine that the information to be flowed is the text on the image, or it can determine that the information to be flowed is the image itself, and can be flexibly adjusted according to the actual application.

[0106] After determining the information to be transferred, the electronic device can generate a drag shadow for that information. If the information is an image, the electronic device can generate a drag shadow based on the bounding rectangle of the image. If the information is text, the electronic device can generate a drag shadow based on the bounding polygon of the text. Optionally, the electronic device can also fill the text to obtain a corresponding fill shape, and then generate a drag shadow based on that fill shape.

[0107] Optionally, the size of the drag shadow of the information to be transferred can be the same as or different from the size of the bounding rectangle, the bounding polygon, or the size of the filled shape corresponding to the information to be transferred. For example, if the information to be transferred is an image, the size of the drag shadow of the image can be the same as the size of the image, or it can be the size of the image after being enlarged or reduced. This application does not limit this.

[0108] S604, in response to detecting a drag operation acting on the drag shadow, determine at least one candidate application that matches the intent type corresponding to the information to be transferred.

[0109] In this embodiment, the intent type corresponding to the information to be transferred can be used to represent the user's intent or purpose in dragging the information to be transferred. Different intent types can represent different user needs. The intent type corresponding to the information to be transferred in this application can also be described as the intent category corresponding to the information to be transferred. Optionally, the intent type may include, but is not limited to, querying information, navigation, searching, social interaction, education and learning, purchasing goods, and booking services.

[0110] In some embodiments, the electronic device can input the information to be transferred into an intent recognition model to obtain the intent type corresponding to the information to be transferred, as output by the intent recognition model. Here, the intent recognition model can be a single model or a combination of multiple models. For example, if the information to be transferred is text, the electronic device can first input the text into an NLU model to obtain the entity type of the text output by the NLU model, and then input the entity type of the text into a prediction model to obtain the intent type output by the prediction model. Optionally, the entity type of the text can include address, movie / TV show, music, time, name, website, phone number, email address, etc. For example, assuming the text content is "No. 100, Street C, District B, City A", the electronic device inputs this text into an NLU model and obtains that the entity type of the text output by the NLU model is address. Then, through the prediction model, it can be determined that the user's intent to drag the text may be navigation. If the information to be transferred is an image, the electronic device can first input the image into a CV model to obtain the image label type output by the CV model, and then input the image label type into a prediction model to obtain the intent type output by the prediction model. Optionally, the image tag types can include animals, plants, vehicles, daily necessities, people, clothing, etc.

[0111] Once an electronic device predicts the intent type corresponding to the information to be transmitted, it can determine at least one candidate application that matches that intent type. For example, if the intent type of the information to be transmitted is navigation, then a candidate application matching navigation could be a map application. Similarly, if the intent type of the information to be transmitted is education and learning, then a candidate application matching education and learning could be an online course application or a digital library application, etc.

[0112] S605, outputs a flow option bar on the first user interface, the flow option bar including an icon of at least one application to be selected.

[0113] In this embodiment, the browsing interface and flow option bar of the first application can be displayed in the same user interface (i.e., the first user interface). Optionally, the flow option bar can be displayed side by side with the browsing interface of the first application, for example, on the left or right side of the browsing interface of the first application. Alternatively, the flow option bar can also be displayed floating on the browsing interface of the first application; this application is not limited to this.

[0114] In some embodiments, the electronic device may display animation effects when outputting the flow option bar on the first user interface. Taking the flow option bar and the browsing interface of the first application side by side as an example, the interface of the first application may, for example, change size as the flow option bar is displayed, or the interface of the first application may also change according to a preset trajectory as the flow option bar is displayed.

[0115] In some embodiments, the icons of at least one candidate application included in the flow options bar can be arranged in descending order of application priority. The application priority can be determined based on user habits; for example, applications used more frequently by the user can have a higher priority. For instance, assuming the information to be flowed is the text "Titanic," the electronic device can recognize the entity type of the text "Titanic" as a movie title, and thus predict that the intent type corresponding to the movie title might be watching a movie or listening to music. Applications matching the intent type corresponding to this text can include video applications, music applications, movie ticket purchasing applications, etc. If the user uses the video application more frequently than the movie ticket purchasing application, and the movie ticket purchasing application more frequently than the music application, then the video application can be considered to have a higher priority than the movie ticket purchasing application, and the movie ticket purchasing application has a higher priority than the music application. Therefore, the flow options bar can sort the application icons in the order of video application, movie ticket purchasing application, and music application.

[0116] S606, based on the stop position of the drag operation in the flow options bar, determine a second application from at least one candidate application.

[0117] In this embodiment, the stopping position of the drag operation in the flow option bar can be understood as the position where the drag shadow is released after being moved to the flow option bar. The drag operation may stop at the location of an application's icon, in which case that application can be identified as the application from which the information to be flowed (i.e., the second application).

[0118] S607 controls the flow of information to be transferred to the second application.

[0119] In some embodiments, after the electronic device controls the flow of information to be transferred to the second application, it can display the user interface of the second application in full screen, or it can display the user interface of the second application as a floating window on the browsing interface of the first application, or it can display the user interface of the second application in a split-screen format. Optionally, the electronic device can display the information to be transferred in the user interface of the second application. For example, assuming the first application is a social application and the second application is a shopping application, and the information to be transferred is an image in the browsing interface of the social application, after the electronic device controls the flow of the image from the social application to the shopping application, it can jump to the product display interface of the shopping application and display the image and the product matching the image in the product display interface of the shopping application. Optionally, the electronic device can also directly display the window corresponding to the information to be transferred in the user interface of the second application. For example, the first application could be a chat application, and the second application could be a short video application. The information to be transferred could be text containing a sharing password in the chat application. This sharing password can be used to open the corresponding short video in the short video application. After the electronic device controls the text containing the sharing password to flow to the short video application, it can jump to the user interface of the short video application to play the short video corresponding to the sharing password.

[0120] In some embodiments, after the electronic device controls the flow of information to be transferred to the second application, it can continue to display the browsing interface of the first application and display a prompt message in the browsing interface of the first application. This prompt message is used to indicate that the flow of information to be transferred has been successful. For example, assuming the first application is a news application and the second application is a bookmarking application, and the information to be transferred is text in the browsing interface of the news application, after the electronic device controls the flow of text in the news application to the bookmarking application, it can continue to display the browsing interface of the news application and display a prompt message in the browsing interface of the news application. For example, the prompt message may include the text "Bookmarked successfully".

[0121] In this embodiment, the electronic device can automatically take a screenshot of the browsing interface of the currently displayed first application after detecting a user's flow trigger operation. It then identifies the information to be flowed in the browsing interface based on the location information of the flow trigger operation, and finally flows the information to be flowed to other applications based on the user's drag-and-drop operation. Compared to methods that require users to manually select the information to be flowed or manually take screenshots for information flow, this embodiment can automate and intelligently transfer information, thus improving the efficiency of information flow between applications.

[0122] The information flow method of this application embodiment will be introduced below with specific scenario examples.

[0123] Example 1, please see Figures 7A-7D , Figures 7A-7D This is a schematic diagram of an information flow method provided in an embodiment of this application. Figures 7A-7D Taking the first application as a chat application, the information to be transferred as text, and the transfer trigger operation as a long press operation, and the second application as a calendar reminder application as an example.

[0124] like Figure 7A As shown, when the electronic device displays the chat interface of a chat application, it detects a long press operation performed by the user on the chat interface and can automatically take a screenshot of the chat interface. By recognizing the screenshot of the chat interface, the electronic device can determine that the candidate information in the chat interface includes text 701 and text 702. Based on the vertex positions of the bounding polygon corresponding to text 701 and text 702, as well as the position information of the long press operation, the electronic device can determine that the long press operation is within the bounding polygon corresponding to text 701, and thus determine that the information the user expects to transfer is text 701.

[0125] like Figure 7B As shown, the electronic device can output a drag-and-drop shadow corresponding to text 701 in the chat interface. When the electronic device detects a user's drag-and-drop operation on the shadow, it can identify at least one candidate application that matches the intent type corresponding to text 701. Specifically, by inputting text 701 into an NLU model, the electronic device can obtain the entity type of text 701, including address, time, and date. Therefore, it can predict that the intent type corresponding to text 701 might be navigation, booking services, search, or memo information. At least one candidate application matching the intent type of text 701 could include map applications, ride-hailing applications, calendar reminder applications, search applications, hotel booking applications, sharing applications, etc.

[0126] like Figure 7CAs shown, the electronic device can display a navigation options bar on the right side of the screen, which can be displayed side-by-side with the chat interface of the first application. The navigation options bar includes icons for a map application, a ride-hailing application, a calendar reminder application, a search application, and a hotel booking application. Users can drag the shadow towards the icon of the application they wish to navigate to. When the electronic device detects that the drag stops near the calendar reminder application icon, it determines that the calendar reminder application is the application the user wishes to navigate to. The electronic device can then control the text 701 to flow to the calendar reminder application.

[0127] like Figure 7D As shown, the electronic device can display a floating window 711, in which the user interface of a schedule reminder application is displayed, and the user interface of the schedule reminder application displays text 701, including schedule reminders created based on text 701.

[0128] Example 2, please see Figures 8A-8D , Figures 8A-8D This is a schematic diagram of an interface for another information flow method provided in an embodiment of this application. Figures 8A-8D Taking the first application as a social application, the information to be transferred as an image, and the transfer trigger operation as a long press operation, and the second application as a shopping application as an example.

[0129] like Figure 8A As shown, when an electronic device displays the browsing interface of a social application, it detects a long-press operation performed by the user on the browsing interface and can automatically take a screenshot of the browsing interface. By recognizing the screenshot of the browsing interface, the electronic device can determine that the candidate information in the browsing interface includes text 801 and image 802. Based on the vertex positions of the bounding polygon corresponding to text 801, the vertex positions of the bounding rectangle corresponding to image 802, and the position information of the long-press operation, the electronic device can determine that the long-press operation is within the bounding rectangle corresponding to image 802, and thus determine that the information the user expects to transfer is image 802.

[0130] like Figure 8B As shown, the electronic device can output the drag shadow corresponding to image 802 on the browsing interface. When the electronic device detects a user's drag operation on the drag shadow, it can determine at least one candidate application that matches the intent type corresponding to image 802. Specifically, by inputting image 802 into a CV model, the electronic device can obtain the tag type of image 802, which includes "people" and "clothing." Therefore, it can predict that the intent type corresponding to image 802 might be clothing purchase, information sharing, memo information, or search. At least one candidate application matching the intent type corresponding to image 802 could include shopping applications, chat applications, collection applications, memo applications, search applications, etc.

[0131] like Figure 8C As shown, the electronic device can display a navigation options bar on the right side of the screen. This navigation options bar includes icons for shopping apps, sharing apps, chat apps, search apps, and favorite apps. When the electronic device detects that the drag operation has stopped near the shopping app icon, it can determine that the shopping app is the app the user wants to navigate to. The electronic device can then control image 802 to navigate to the shopping app.

[0132] like Figure 8D As shown, the electronic device can jump to the product display interface of the shopping application, which displays image 802 and the products associated with image 802.

[0133] It should be noted that the user interface of the second application (such as the schedule reminder application in Example 1 and the shopping application in Example 2) in the embodiments of this application can be displayed in full screen on the display screen, or it can be displayed in a floating window or split screen mode. This application will not provide examples of each one.

[0134] The implementation process of the information flow method in this application embodiment will be described below from the perspective of module interaction in electronic devices.

[0135] Please see Figure 9 , Figure 9 This is a flowchart illustrating another information flow method provided in an embodiment of this application. Figure 9 As shown, this information flow method includes, but is not limited to, the following steps:

[0136] S901 displays the browsing interface of the first application.

[0137] The browsing interface of the first application can be understood as the interface used to display information within the first application. This information includes, but is not limited to, text, images, videos, audio, or other media types. For an example, please refer to [link to example]. Figure 8A , Figure 8A The interface shown can be understood as the browsing interface of a social application.

[0138] S902 receives the user's flow trigger operation on the browsing interface of the first application.

[0139] As described in the foregoing embodiments, the flow triggering operation can be a long press, double-tap the screen, single-tap the screen, knuckle selection, or other preset operations. Optionally, after the electronic device detects a flow triggering operation, it can distribute the flow triggering operation event to the corresponding module based on Android's AMS, WMS, and View framework.

[0140] S903 triggers an information flow event.

[0141] For example, taking a long press as the trigger action for the flow, the content recognition module can receive touch events from the View framework and set a timer. If the duration of the touch event exceeds the timer's time threshold, the information flow event can be triggered.

[0142] S904, the content recognition module obtains a screenshot image corresponding to the browsing interface of the first application, as well as the location information of the flow trigger operation.

[0143] The content recognition module can automatically capture the browsing interface of the first application and obtain the location information of the flow trigger operation in the browsing interface.

[0144] S905, the content recognition module sends a screenshot image and the location information of the flow trigger operation to the content recognition server.

[0145] The content recognition module can transmit the acquired screenshot image and the location information of the flow trigger operation to the content server. Correspondingly, the content server can receive the screenshot image and the location information of the flow trigger operation from the content recognition module.

[0146] S906, the content recognition server sends a service request to the visual middleware platform.

[0147] The content recognition server can send server requests to the visual platform. Correspondingly, the visual platform can receive service requests from the content server. Service requests can be used to request CV (CV) services for recognizing screenshot images.

[0148] S907, the visual middleware sends a service response to the content recognition server.

[0149] If the visual platform agrees to provide CV services, it can send a service response to the content recognition server.

[0150] S908, the content recognition server sends a screenshot image to the visual middleware platform.

[0151] After receiving the service response, the content recognition server can send a screenshot image to the visual platform.

[0152] S909, the visual platform identifies the content of a screenshot image and obtains the location information of candidate information in the screenshot image.

[0153] The visual platform can identify text or images in screenshots and determine the location information of the identified text or images. Optionally, the location information of text can be the vertex positions of the bounding polygon determined according to the dimensions of blocks, paragraphs, or lines. The location information of images can be the positioning position of the bounding rectangle corresponding to the image.

[0154] S910, the visual middleware platform sends the location information of candidate information in the screenshot image to the content recognition server.

[0155] The visual middleware platform can send the recognized text and / or images, along with the text's location information and / or the image's location information, to the content recognition server.

[0156] Optionally, after receiving the recognition results from the visual platform, the content recognition server can instruct the visual platform to temporarily shut down the CV service, thereby reducing system resource consumption.

[0157] S911, the content recognition server determines the information to be transferred from the screenshot image based on the location information of the transfer trigger operation.

[0158] Based on the location information of the flow trigger operation, the location information of the text in the screenshot image provided by the visual platform, and / or the location information of the image, the content recognizer can determine the information to be flowed in the screenshot image. For example, the content recognizer can determine the information to be flowed according to the strategy described in Table 1 below.

[0159] Table 1

[0160]

[0161] It should be noted that strategy 3 in Table 1 is merely an illustrative example. If the flow trigger operation is located both within the bounding polygon of the text and the bounding rectangle of the image, the information to be flowed can also be determined to be an image. The specific approach can be adjusted according to the actual situation, and this application does not limit this. Furthermore, regarding strategy 5, images that are too small are usually decorative elements in the interface, and their content has no practical meaning; therefore, they can be filtered out.

[0162] S912, the content recognition server sends the information to be transferred to the content recognition module.

[0163] After the content recognition server determines the information that the user expects to be transferred, it can send the information to the content recognition module. Correspondingly, the content recognition module can receive the information from the content recognition server.

[0164] S913, the content recognition module generates a drag shadow for the information to be transferred.

[0165] The content recognition module can generate corresponding drag shadows based on the type of information to be transferred. For images, the module can further crop the image based on its bounding rectangle and then generate a drag shadow based on the cropped image. For text, the module can fill a TextView (text style) according to preset rules and generate a drag shadow based on the filled TextView.

[0166] S914 outputs a drag shadow of the information to be transferred on the browsing interface of the first application.

[0167] After the content recognition module generates the drag shadow of the information to be transferred, it can output the drag shadow of the information to be transferred on the browsing interface of the first application.

[0168] S915 receives drag operations from the user that affect the drag shadow.

[0169] S916, triggers a drag-and-drop event.

[0170] Among them, the Android-based drag-and-drop framework can monitor the user's drag-and-drop operations and then trigger drag-and-drop events.

[0171] S917, the content recognition module sends the information to be transferred to the intelligent transfer module.

[0172] The content recognition module can invoke the drag-and-drop frame capability to trigger the smart workflow module to start the smart workflow service and send the workflow information to be workflowd to the smart workflow module. Correspondingly, the smart workflow module can receive the workflow information from the content recognition module.

[0173] S918, the intelligent flow module sends the information to be flowed to the intent platform.

[0174] The intelligent flow module can transmit the information to be flowed to the intent platform so that the intent platform can identify the intent type corresponding to the information to be flowed.

[0175] S919, the intent platform sends text to the NLU platform.

[0176] If the information to be transferred is text, the intent platform can send text to the NLU platform.

[0177] S920, the NLU platform inputs text into the NLU model and obtains the entity type of the text.

[0178] Among them, the NLU platform can identify the entity type of text through the NLU model.

[0179] S921, the entity type of the text sent by the NLU middle platform to the intent middle platform.

[0180] The NLU platform can send the entity types of the identified text to the intent platform. Correspondingly, the intent platform can receive the entity types of the text from the NLU platform.

[0181] S922, the intent platform sends images to the visual platform.

[0182] When the information to be transferred is an image, the intent platform can send the image to the visual platform.

[0183] S923, the visual platform inputs images into the CV model and obtains the image's label type.

[0184] Among them, the visual middleware platform can identify the label type of an image using a CV model.

[0185] S924, the tag type of the image sent by the visual middle platform to the intent middle platform.

[0186] The visual middleware platform can send the label type of the identified image to the intent middleware platform. Correspondingly, the intent middleware platform can receive the label type of the image from the visual middleware platform.

[0187] S925, the intent platform predicts the intent type of the information to be transferred based on the entity type of the text and / or the tag type of the image.

[0188] The intent platform can input the entity type of text and / or the label type of image into the prediction model to obtain the intent type output by the prediction model.

[0189] S926, the intent platform sends a query request to the service management platform.

[0190] After obtaining the intent type of the information to be transferred, the intent platform can send a query request to the service management platform. The query request can be used to query at least one application (or service) that matches the intent type.

[0191] S927, the service management platform sends a query response to the intent platform.

[0192] The service management platform can send query responses to the intent platform. The query response can include at least one application that matches the intent type.

[0193] S928, the intent platform sorts applications according to application priority, and obtains a sorted application list.

[0194] The intent platform can sort applications according to their priority from high to low, resulting in an ordered list of applications.

[0195] S929, the intent platform sends the application list to the intelligent flow module.

[0196] The intent platform can send an ordered list of applications to the intelligent workflow module. Correspondingly, the intelligent workflow module can receive an ordered list of applications from the intent platform.

[0197] S930, Intelligent Flow Module Control Output Flow Options Bar.

[0198] The intelligent workflow module can control the display screen to output a workflow option bar, which can include icons of an ordered list of applications.

[0199] S931, The user's drag operation was detected and stopped when the user moved to the icon of the second application.

[0200] S932, the intelligent transfer module controls the transfer of information to be transferred to the second application.

[0201] If the system detects that the user's drag operation stops near the icon of the second application, it can be determined that the user expects the information to flow to the second application. At this point, the intelligent flow module can control the flow of the information to be flowed to the second application.

[0202] By implementing this application, an electronic device can automatically take a screenshot of the browsing interface of the currently displayed first application after detecting a user's flow trigger operation. Based on the location information of the flow trigger operation, it can identify the information to be flowed in the browsing interface, and then, based on the user's drag operation and the detected drag operation, flow the information to be flowed to other applications. This automates and intelligently processes information flow, improving the efficiency of information flow between applications.

[0203] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive).

[0204] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0205] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0206] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0207] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: flash drive, ROM, RAM, disk or optical disk, etc.

[0208] The above-disclosed embodiments are merely one preferred embodiment of this application and only a part of the embodiments of this application. They should not be construed as limiting the scope of the claims of this application.

Claims

1. A method of information flow conversion, characterized by, The method comprises: displaying a first user interface, the first user interface comprising a browsing interface of a first application; in response to detecting a flow triggering operation acting on the browsing interface, obtaining a screenshot image corresponding to the browsing interface; determining, from the screenshot image, information to be flowed according to position information of the flow triggering operation, and outputting a drag shadow of the information to be flowed on the browsing interface; in response to detecting a drag operation acting on the drag shadow, determining at least one candidate application matching an intent type corresponding to the information to be flowed; outputting a flow selection bar on the first user interface, the flow selection bar comprising icons of the at least one candidate application; determining a second application from the at least one candidate application based on a stop position of the drag operation in the flow selection bar; controlling the information to be flowed to flow to the second application.

2. The method of claim 1, wherein, The method comprises: identifying position information of candidate information in the screenshot image; determining, from the candidate information, information to be flowed according to the position information of the candidate information in the screenshot image and the position information of the flow triggering operation.

3. The method of claim 2, wherein, The candidate information comprises at least one picture, and the position information of one picture comprises vertex positions of a circumscribed rectangle corresponding to the one picture. The method comprises: determining that the flow triggering operation is within a first circumscribed rectangle according to the vertex positions of the circumscribed rectangle corresponding to the at least one picture and the position information of the flow triggering operation; determining, as the information to be flowed, a picture corresponding to the first circumscribed rectangle.

4. The method of claim 3, wherein, The method further comprises: generating a drag shadow of the information to be flowed based on the picture corresponding to the first circumscribed rectangle.

5. The method of claim 2, wherein, The candidate information comprises at least one piece of text, and the position information of one piece of text comprises vertex positions of a circumscribed polygon corresponding to the one piece of text. The method comprises: determining that the flow triggering operation is within a first circumscribed polygon according to the vertex positions of the circumscribed polygon corresponding to the at least one piece of text and the position information of the flow triggering operation; determining, as the information to be flowed, a piece of text corresponding to the first circumscribed polygon.

6. The method of claim 2, wherein, The candidate information comprises at least one picture and at least one piece of text, the position information of one picture comprises vertex positions of a circumscribed rectangle corresponding to the one picture, and the position information of one piece of text comprises vertex positions of a circumscribed polygon corresponding to the one piece of text. The method comprises: determining, from the candidate information, information to be flowed according to the position information of the candidate information in the screenshot image and the position information of the flow triggering operation. determine, according to a position of a vertex of an outer rectangle corresponding to the at least one picture, a position of a vertex of an outer polygon corresponding to the at least one piece of text, and position information of the flow triggering operation, that the flow triggering operation is within the second outer rectangle and within the second outer polygon; determine text corresponding to the second outer polygon as the information to be flowed.

7. The method according to any one of claims 1 to 6, wherein The method further includes: input the information to be flowed into an intent recognition model to obtain an intent type corresponding to the information to be flowed output by the intent recognition model.

8. The method of claim 1, wherein, The icons of the at least one candidate application are sorted in descending order of application priority.

9. The method according to any one of claims 1 to 8, wherein, The flow triggering operation is a pressing operation with a duration greater than or equal to a time threshold.

10. The method of any one of claims 1-9, wherein, The method further includes: jump from the first user interface to a user interface of the second application, and the user interface of the second application displays the information to be flowed.

11. An electronic device, comprising: include a memory and one or more processors; the memory is coupled with the one or more processors, and is used to store a computer program including program instructions; the one or more processors invoke the program instructions, so that the electronic device executes the method according to any one of claims 1-10.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1-10.

13. A chip system, characterized by The chip system is coupled with the memory, and the chip system is used to read and execute the computer program stored in the memory to implement the method according to any one of claims 1-10. The chip system is coupled with the memory, and the chip system is used to read and execute the computer program stored in the memory to implement the method according to any one of claims 1-10.

Images