Information transmission method, electronic equipment and storage medium

Abstract

The embodiment of the invention relates to the technical field of terminals, in particular to an information transmission method, electronic equipment and a storage medium. In the method, the electronic equipment can perform page analysis on the screenshot of the electronic equipment when the user triggers the selection operation to obtain the page analysis result. And then, the electronic equipment determines a transmission object according to the page analysis result and the trajectory of the selection operation. And then, the electronic equipment displays a service triggering interface according to the transmission object. In the method, the electronic equipment determines the transmission object through the page analysis result and the trajectory of the selection operation, so that the flexibility of the display service of the electronic equipment can be improved, and the use experience of a user is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of terminal, and in particular, relate to a method for transmitting information, an electronic device and a storage medium. BACKGROUND

[0002] A service, also referred to as an atomic service, an application service, an application microservice, etc., refers to a program entity that can be executed independently and implements a preset function. The electronic device can display a plurality of services on a service triggering interface according to the transmitted information, and the user can trigger a service through the interface to make the electronic device execute the preset function of the service triggered by the user.

[0003] At present, how to transmit information to make the electronic device display services more flexibly is a problem to be solved. SUMMARY

[0004] Embodiments of the present application provide a method for transmitting information, an electronic device and a storage medium, and the information transmitted by the method can make the electronic device display services flexibly.

[0005] To achieve the above object, embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, a method for transmitting information is provided, which is applied to an electronic device such as a mobile phone or a tablet computer, and the method comprises: the electronic device displays a first interface. Then, the electronic device receives a first selection operation acting on the first interface, and the first selection operation is a circle selection operation. Then, in response to the first selection operation, the electronic device displays a circle selection track corresponding to the first selection operation and obtains a screenshot of the first interface. Next, the electronic device performs page analysis on the screenshot to obtain a page analysis result, and the page analysis result includes a text region and / or an image region in the first interface. Then, the electronic device determines a transmission object according to the circle selection track and the page analysis result. Next, after the electronic device determines the transmission object, the electronic device displays a second interface, and the second interface includes: a deformed screenshot, the transmission object marked on the deformed screenshot, and a service icon of a service corresponding to the transmission object.

[0007] In this method, the transmission object is determined according to the circle selection track and the page analysis result, that is, the electronic device can determine the transmission object in combination with the content of the page and the operation of the user. In this way, the matching degree of the transmission object with the operation of the user will be improved, and the matching degree of the transmission object with the content of the page will also be improved. This will make the subsequent display of services by the mobile phone more flexible and improve the user experience.

[0008] In a possible design of the first aspect, the electronic device determines the transmission object according to the circle selection track and the page analysis result, including: the electronic device obtains a selection range according to the circle selection track. The selection range represents the content selected by the first selection operation on the first interface. Then, the electronic device matches the selection range and the page analysis result to obtain the transmission object.

[0009] In this design, the electronic device can quickly and accurately obtain the transmission object by matching the selection range and the page analysis result.

[0010] In another possible design of the first aspect, the page analysis result includes at least one text region and at least one image region. The electronic device matches the selection range and the page analysis result to obtain the transmission object, including: the electronic device obtains a target text region that has an intersection with the selection range in the at least one text region. Then, the electronic device obtains a ratio of the intersection part of the selection range and the target text region to the target text region. Next, in a case where the ratio is greater than or equal to a first threshold, the electronic device obtains a part other than the intersection part in the selection range. Then, the electronic device determines the transmission object according to the area of the intersection part and the area of the part other than the intersection part.

[0011] In this design, the electronic device can determine the transmission object according to the area of the intersection part and the area of the part other than the intersection part, which can further improve the matching degree of the transmission object and the page content, and further improve the matching degree of the transmission object and the user operation. Therefore, the flexibility of the mobile display service can be further improved, and the user experience can be improved.

[0012] In yet another possible design of the first aspect, the electronic device determines the transmission object according to the area of the intersection part and the area of the part other than the intersection part, including: in a case where the area of the part other than the intersection part is greater than the area of the intersection part, the electronic device determines that the transmission object is an image included in the selection range.

[0013] In this design, the electronic device can determine that the transmission object is an image included in the selection range in a case where the area of the part other than the intersection part is greater than the area of the intersection part. Therefore, the flexibility of the mobile display service can be further improved, and the user experience can be improved.

[0014] In an example of the first aspect, before determining the transmission object as the image included in the selection range, the method further includes: obtaining, by the electronic device, a target image region in the at least one image region that has an intersection with the selection range. Then, in a case where a ratio of the intersection region of the selection range and the target image region to the target image region is greater than or equal to a second threshold, the electronic device merges the target image region and the selection range to obtain a first merged region. Then, the electronic device determines the transmission object as the image included in the first merged region. In addition, in a case where the ratio of the intersection region of the selection range and the target image region to the target image region is less than the second threshold, the electronic device determines the transmission object as the image included in the selection range.

[0015] In this example, the electronic device can determine different transmission objects according to different ratios of the intersection region of the selection range and the target image region to the target image region. Thus, the flexibility of the mobile display service can be further improved, and the user experience can be improved.

[0016] In another example of the first aspect, the determining the transmission object according to the area of the intersection part and the area of the other part includes: in a case where the area of the intersection part is greater than the area of the other part, merging, by the electronic device, the selection range and the target text region to obtain a second merged region. Then, the electronic device determines the text included in the second merged region as the transmission object.

[0017] In this example, the electronic device can determine the text included in the second merged region as the transmission object in a case where the area of the intersection part is greater than the area of the other part. Thus, the flexibility of the mobile display service can be further improved, and the user experience can be improved.

[0018] In another example of the first aspect, the matching the selection range and the page analysis result to obtain the transmission object further includes: in a case where the ratio is less than the first threshold, determining, by the electronic device, the transmission object as the image included in the selection range.

[0019] In this example, the electronic device can determine the transmission object as the image included in the selection range in a case where the ratio is less than the first threshold. Thus, the flexibility of the mobile display service can be further improved, and the user experience can be improved.

[0020] In another example of the first aspect, the obtaining the selection range according to the circling track includes: obtaining, by the electronic device, a circumscribed regular polygon of the track data according to the track data corresponding to the circling track. Then, the electronic device takes the circumscribed regular polygon as the selection range.

[0021] In this design, electronic devices can easily obtain a selection range by using an external regular polygon.

[0022] In another possible design of the first aspect, the second interface further includes a first control. The method further includes: the electronic device receiving a first trigger operation applied to the first control. Then, in response to the first trigger operation, the electronic device updates the type of the transmitted object from a first type to a second type, where the first type differs from the second type, and the type of the transmitted object includes text or image. Next, the electronic device displays a third interface, which includes: a distorted screenshot, the updated transmitted object identified on the distorted screenshot, and a service icon corresponding to the updated transmitted object.

[0023] In this design, the electronic device can update the transmitted object through a first control, such as changing the type of the transmitted object from a first type to a second type. This further enhances the flexibility of mobile phone display services and improves the user experience.

[0024] In another possible design of the first aspect, the above method further includes: when the transmitted object is an image that does not contain text, in response to a triggering operation, the electronic device displays a prompt indicating that the transmitted object does not contain text.

[0025] In another possible design of the first aspect, the second interface further includes a second control. The method further includes: the electronic device receiving a second trigger operation applied to the second control. Then, in response to the second trigger operation, the electronic device displays an editing interface for the transmitted object.

[0026] In this design, users can also trigger editing of the transmitted object through a second control. This improves the user experience.

[0027] In another possible design of the first aspect, the second interface further includes a third control. The method also includes: the electronic device receiving a third trigger operation applied to the third control. Subsequently, in response to the third trigger operation, the electronic device stores the transmitted object in a preset location.

[0028] In this design, users can also trigger the electronic device to store the transmitted object via a third-party control. This enhances the user experience.

[0029] In another possible design of the first aspect, the above-mentioned page analysis of the screenshot to obtain the page analysis result includes: the electronic device processes the screenshot through a computer vision model to obtain at least one text region and at least one image region.

[0030] In another possible design of the first aspect, the service icon of the service corresponding to the aforementioned transmission object includes the service icon of the target service. The method further includes: in response to a triggering operation on the service icon of the target service, the electronic device displays the interface of the application corresponding to the target service.

[0031] In a second aspect, an electronic device is provided, the electronic device including a memory and one or more processors, the memory being coupled to the processors; wherein the memory stores computer program code, the computer program code including computer instructions; when the computer instructions are executed by the processor, the electronic device performs the method provided by the first aspect and any possible design of the first aspect.

[0032] Thirdly, a computer-readable storage medium is provided, including computer instructions that, when executed on an electronic device, cause the electronic device to perform the methods provided by the first aspect and any possible design of the first aspect.

[0033] Fourthly, a computer program product containing instructions is provided, which, when run on an electronic device, enables the electronic device to perform the methods provided by the first aspect and any possible design of the first aspect.

[0034] Fifthly, a chip system is provided for use in an electronic device, the chip system including one or more processors for invoking computer instructions to cause the electronic device to perform the methods provided in the first aspect and any possible design of the first aspect.

[0035] The technical effects of any of the design methods in aspects two through five can be found in the technical effects of different design methods in aspect one, and will not be repeated here. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0037] Figure 2 A schematic diagram of the architecture of an electronic device provided in an embodiment of this application;

[0038] Figure 3 This is a schematic diagram of the architecture of any gate-related applications in electronic devices provided in the embodiments of this application;

[0039] Figure 4 A flowchart illustrating the information transmission method provided in this application embodiment;

[0040] Figure 5 A schematic diagram illustrating page analysis provided for embodiments of this application;

[0041] Figure 6 A set of schematic diagrams showing page analysis results provided for embodiments of this application;

[0042] Figure 7 Another set of schematic diagrams showing page analysis results provided for embodiments of this application;

[0043] Figure 8 A schematic diagram of a set of user image interfaces provided in the embodiments of this application;

[0044] Figure 9 A schematic diagram of another set of user image interfaces provided in the embodiments of this application;

[0045] Figure 10 A schematic diagram of yet another set of user image interfaces provided in the embodiments of this application;

[0046] Figure 11 A schematic diagram of another set of user image interfaces provided in the embodiments of this application;

[0047] Figure 12 A schematic diagram of yet another set of user image interfaces provided in the embodiments of this application;

[0048] Figure 13 A schematic diagram of another set of user image interfaces provided in the embodiments of this application;

[0049] Figure 14 A schematic diagram of yet another set of user image interfaces provided in the embodiments of this application;

[0050] Figure 15 A schematic diagram of another set of user image interfaces provided in the embodiments of this application;

[0051] Figure 16 A schematic diagram of yet another set of user image interfaces provided in the embodiments of this application;

[0052] Figure 17 A schematic diagram of another set of user image interfaces provided in the embodiments of this application;

[0053] Figure 18 A schematic diagram of yet another set of user image interfaces provided in the embodiments of this application;

[0054] Figure 19 A schematic diagram of another set of user image interfaces provided in the embodiments of this application;

[0055] Figure 20 A schematic diagram of yet another set of user image interfaces provided in the embodiments of this application;

[0056] Figure 21 This is a schematic diagram of the structure of another electronic device provided in an embodiment of this application;

[0057] Figure 22 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0058] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of this application, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can represent A or B. "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. Furthermore, in the description of the embodiments of this application, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. Furthermore, to facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0059] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being better or more advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner to facilitate understanding.

[0060] The technical solutions disclosed in this application involve the collection, storage, use, processing, transmission, provision, and disclosure of users' personal information, all of which comply with relevant laws and regulations and do not violate public order and good morals.

[0061] A service, also known as an atomic service, application service, application microservice, etc., refers to a program entity that can execute independently and perform a pre-defined function. These pre-defined functions can include things like hailing a ride, ordering food, navigation, setting an alarm, setting a memo, etc. Services can be implemented in various forms such as Android applications, quick apps, mini-programs, and web services. Services are characterized by independence, composability, scalability, and substitutability.

[0062] Currently, electronic devices can determine the transmission target based on a selection action (e.g., long-press selection). The device then displays a service trigger interface based on the transmission target. However, during this process, the electronic device can directly use the selected content as the transmission target. This results in inflexible service display on the trigger interface, failing to meet user experience requirements.

[0063] In view of this, embodiments of this application provide an information transmission method. In this method, an electronic device can perform page analysis on a screenshot of the screen taken when a user triggers a selection operation, and obtain page analysis results. Then, the electronic device determines the transmission target based on the page analysis results and the trajectory of the selection operation. Finally, the electronic device displays a service trigger interface based on the transmission target.

[0064] In this method, electronic devices determine the transmission object based on page analysis results and the trajectory of selection operations, which can improve the flexibility of electronic device display services and enhance the user experience.

[0065] It is understood that the technical solutions provided in this application can be applied to electronic devices with display functions. These electronic devices can also be referred to as terminals, terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. These electronic devices can be mobile phones, tablets, wearable devices, smart screens, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and other electronic devices with interface display functions. This application does not impose any limitations on the product form of the electronic devices.

[0066] The structure and architecture of electronic devices will be introduced below.

[0067] Figure 1 A schematic diagram of the hardware structure of the electronic device 100 is shown.

[0068] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, sensor 180, and display screen 194, etc. Among them, sensor 180 may include pressure sensor 180A, touch sensor 180K, etc.

[0069] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0070] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. The different processing units may be independent devices or integrated into one or more processors.

[0071] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0072] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0073] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0074] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL).

[0075] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194. The MIPI interface includes a display serial interface (DSI). The processor 110 and the display screen 194 communicate through the DSI interface to realize the display function of the electronic device 100.

[0076] USB interface 130 is an interface that conforms to the USB standard specification, specifically it can be a Mini USB interface, Micro USB interface, USB Type C interface, etc.

[0077] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0078] Electronic device 100 implements display functions through display screen 194 and application processor, etc. GPU is a microprocessor for image processing, connected to display screen 194 and application processor. GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0079] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0080] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0081] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0082] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the intensity of the touch operation based on pressure sensor 180A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0083] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch screen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0084] After introducing the hardware structure of electronic devices, we will now introduce the software architecture of electronic devices.

[0085] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses a layered architecture of Android. TM Taking the system as an example, the software architecture of electronic device 100 is illustrated.

[0086] Figure 2 This is a schematic diagram of the architecture of an electronic device 100 according to an embodiment of the present invention.

[0087] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0088] The application layer can include a series of application packages.

[0089] like Figure 2 As shown, the application package can include applications such as camera, gallery, calendar, maps, navigation, and SMS. The application package may also include Honor... TM The application package for Honor AnyDoor. The application package may also include an application package for screenshot applications. In the following description, Honor AnyDoor will also be referred to as AnyDoor for short.

[0090] The "Any Door" is used to display the service trigger interface, through which users can trigger services and instruct electronic devices to perform the preset functions of the triggered services.

[0091] Among them, screenshot is used to provide the screenshot function of electronic devices, such as double-tap screenshot, knuckle screenshot, key screenshot, etc.

[0092] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0093] like Figure 2 As shown, the application framework layer may include a window manager, content provider, view system, resource manager, etc.

[0094] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0095] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0096] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0097] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0098] The application framework layer may also include knuckle framework, intent framework, visual middleware, etc.

[0099] The knuckle frame can send knuckle events. For example, the knuckle frame sends a knuckle event after the user triggers a knuckle operation on the display screen. For further information on knuckle operations and knuckle events, please refer to related technologies; this application does not impose any limitations on these aspects.

[0100] The intent framework can receive user intents and obtain services that match those intents.

[0101] The visual platform can be used to implement computer vision (CV) related functions. For example, it can perform text recognition on images to obtain the text contained in the images, or extract image content, etc.

[0102] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0103] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0104] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0105] A system library can include multiple functional modules. For example, a surface manager and media libraries.

[0106] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0107] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0108] The system library may also include computer vision (CV) models and natural language processing (NLP) models.

[0109] Among them, CV (Characteristic Vision) models can perform text recognition on images to obtain the text contained within them. They can also extract image content. NLP (Natural Language Processing) models are used to process natural language and obtain language processing results. For further information on CV and NLP models, please refer to relevant technologies; this application does not impose any limitations on their implementation.

[0110] The kernel layer is the layer between hardware and software. The kernel layer includes at least display drivers, camera drivers, and sensor drivers.

[0111] Below, we will consider the scenario of an electronic device displaying a service trigger interface, combined with the above... Figure 2 This example illustrates the workflow of software in an electronic device.

[0112] After a user triggers a knuckle selection action on an electronic device, the knuckle frame sends a knuckle event to AnyDoor. AnyDoor then determines the selection area based on the knuckle event. Further details regarding the selection area are provided below and will not be repeated here. Next, AnyDoor captures a screenshot of the screen when the user triggers the knuckle selection action and sends it to the visual platform. The visual platform uses a CV model to perform layout recognition on the screenshot, obtaining the text and image regions.

[0113] Next, if the selected area matches a text region, the visual platform uses an NLP model to perform natural language recognition on the text within the text region to obtain text entities. Alternatively, if the selected area matches an image region, the visual platform performs image recognition on the image within the image region to obtain image entities. Then, the portal sends either the image entity or the text entity to the intent framework. The intent framework then identifies the user's intent based on the text entity or the image entity and obtains a list of services corresponding to that intent. Finally, the portal retrieves this service list and displays the service trigger interface based on it.

[0114] It should be understood that any door can adopt a layered architecture. A layered architecture divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces.

[0115] For example, see Figure 3 , Figure 3This diagram illustrates the architecture of an application related to "AnyDoor" for electronic devices, such as the architecture diagram of the AnyDoor application. AnyDoor can be divided into an application layer and a business middleware layer. The application layer includes a selection service, a service loading module, and a service triggering module. The business middleware layer includes a content recognition framework and an intent recognition module.

[0116] Below, we will consider the scenario of an electronic device displaying a service trigger interface, combined with the above... Figure 3 This example illustrates the workflow of software in an electronic device.

[0117] The selection service receives knuckle events and determines the selection area based on these events. Additionally, the selection service captures a screenshot. The process of the selection service receiving knuckle events is as follows: Figure 3 The text is denoted by the symbol ①. A screenshot can be understood as a capture of the screen of an electronic device, including the displayed elements on that screen.

[0118] Next, the selection service sends the screenshot and the selected area to the content recognition framework. The content recognition framework performs layout recognition on the screenshot to obtain the text and image regions. Then, if the selected area matches a text region, the content recognition framework performs natural language processing (NLP) on the text within the selected area to obtain text entities; or, if the selected area matches an image region, the content recognition framework performs image recognition on the image within the selected area to obtain image entities. The process of the content recognition framework acquiring the screenshot and the selected area is described in detail below. Figure 3 It is indicated by the symbol ②.

[0119] Next, the content recognition framework sends the text entity or image entity to the intent recognition module. The intent recognition module then obtains the user intent matched with the text entity, or the user intent matched with the image entity. The process of the content recognition framework sending text entities or image entities to the intent recognition module is described in detail below. Figure 3 It is indicated by the symbol ③.

[0120] Next, the intent recognition module sends the user's intent to the service loading module, which then retrieves the list of services corresponding to the user's intent. Figure 3 This is indicated by the label ④. Then, the electronic device displays the service trigger interface based on the service list.

[0121] Then, in response to the user's trigger operation on the target service, the service loading module calls the service triggering module, causing the electronic device to execute the preset function corresponding to the triggered application. The process of the service loading module calling the service triggering module is as follows: Figure 3 It is indicated by the symbol ⑤.

[0122] Below, we take a mobile phone as an example of an electronic device, which has the above-mentioned features. Figure 1 The structure shown, and Figure 2 Taking the illustrated architecture as an example, the technical solutions provided in the embodiments of this application will be introduced.

[0123] For example, see Figure 4 The information transmission method provided in this application embodiment may include steps S400-S407. Optionally, after step S407, the information transmission method provided in this application embodiment may further include step S408.

[0124] S400. In response to a knuckle selection operation, the phone's knuckle frame sends a knuckle event.

[0125] The knuckle selection operation can be a user's finger knuckle touching the screen and then sliding the knuckle across a designated area on the screen. Knuckle events can include trajectory data, which represents the path the knuckle travels on the screen. For example, the trajectory data can be data of the `jester` class, and can include the coordinates of multiple points.

[0126] As one possible example, with the knuckle switch on, in response to a knuckle selection operation, the phone's knuckle frame sends a knuckle event to the "Anywhere Door" (a device that allows knuckle gestures). Further information about knuckle switches can be found below. Figure 10 The relevant information will not be elaborated here.

[0127] As a possible example, if the Any Door is not running, the knuckle frame can instruct the Any Door to be activated. The knuckle frame then sends a knuckle event to the Any Door, including the aforementioned trajectory data.

[0128] As another possible example, in the case of any door being in operation, the knuckle frame sends the aforementioned knuckle events to any door, the knuckle events including trajectory data.

[0129] As an example, the knuckle frame can identify the user's knuckle selection operation through the phone's sensors, such as touch sensors and pressure sensors, to obtain knuckle events.

[0130] Understandably, in some other examples, the above-described knuckle selection operation can be replaced with other selection operations, such as single-finger selection, selection of a specific input tool (e.g., stylus), etc.; the corresponding generated events and the module that generates the events can be adjusted. For specific details, please refer to related technologies; this application's embodiments are not limited in this regard.

[0131] S401. After receiving the knuckle event, AnyDoor obtains a screenshot and passes it through to the visual middleware.

[0132] A screenshot can be a snapshot of the phone's screen taken when a user triggers a knuckle selection action. A screenshot can be understood as a capture of the screen of an electronic device, including all the display elements of the screen.

[0133] As an example, the screenshot may be obtained by the screenshot module in response to a knuckle selection operation.

[0134] For example, in response to a knuckle selection action, the knuckle frame calls the screenshot module to obtain a screenshot. The knuckle frame then sends the screenshot to AnyDoor.

[0135] Optionally, AnyDoor can also draw trajectory animations on the phone's display screen based on the trajectory data included in the knuckle events.

[0136] S402. The visual middleware platform performs page analysis on the screenshot to obtain the page analysis results of the screenshot.

[0137] The layout analysis results of a screenshot can include both the image area and the text area. As an example, for text areas, the visual platform can perform layout analysis at the line level. That is, one line of text constitutes one text area. A line can be understood as text occupying an entire line, or as text occupying a portion of an entire line. Alternatively, for text areas, the visual platform can perform layout analysis at the paragraph level. That is, a paragraph of text constitutes one text area.

[0138] It should be understood that in some other implementations, the aforementioned visual platform can also perform layout analysis on text areas at the column level. In the following embodiments, the implementation of this application is described using the visual platform performing layout analysis on text areas at the behavior level as an example, and this application is not limited in this respect.

[0139] As an example, for image regions, the visual platform can perform layout analysis on an image-by-image basis. That is, an image is considered an image region. Alternatively, for image regions, the visual platform can perform layout analysis on a content-object basis. Here, a content object can be any object included within the image content. In other words, a content object is considered an image region.

[0140] In one implementation, the visual platform can perform layout analysis on text regions at the line level; and it can also perform layout analysis on image regions at the image level. The visual platform can represent a text region using the coordinates of the rectangle it occupies. The rectangle's coordinates can include functions of the four sides that make up the rectangle. Alternatively, the rectangle's coordinates can include the coordinates of two non-adjacent vertices (e.g., the top-left and bottom-right vertices, or the top-right and bottom-left vertices).

[0141] For example, a visual platform can represent a text region as a function of the four sides of a rectangle occupied by a line of text. Similarly, a visual platform can represent an image region as a function of the coordinates of a rectangle occupied by an image region.

[0142] The rectangle occupied by the text area can be understood as the smallest bounding rectangle of the text area; the rectangle occupied by the image area can be understood as the smallest bounding rectangle of the image area.

[0143] For example, see Figure 5 During the display of the image and text browsing interface on the mobile phone, in response to a knuckle event, the screenshot module performs a screenshot operation to obtain screenshot 500. The aforementioned image and text browsing interface can be a webpage display interface of a browser application; alternatively, it can be an interface displayed by other applications, including image and text elements. Afterwards, the "AnyDoor" acquires screenshot 500 and sends it to the visual platform. The visual platform uses a CV model to perform page analysis on screenshot 500 to obtain page analysis result 510.

[0144] Page analysis results 510 may include: image text areas and line text areas.

[0145] For example, the page analysis result 510 may include an image region 511, and the visual platform can represent the image region 511 as a function of the four sides of the smallest bounding rectangle of the image region 511.

[0146] For example, the page analysis result 510 may include a text region 512. The visual platform can represent the text region 512 by the function values ​​of the four sides of the smallest bounding rectangle of the text region 512.

[0147] It should be understood that the page analysis results may also include text areas such as “AAAAAAAAAAAAAAAAA” and “AAAAAAAAACCAAAAAA”. The visual platform represents these two text areas in a similar way to the above, and will not be elaborated here.

[0148] in, Figure 5The letter "A" in this context can refer to one or more arbitrary letters or characters. Figure 5 The "BB" in the text can be used to refer to the word "pet dog"; Figure 5 The "CC" in the text can be used to refer to the word "park". It should be noted that, regarding the appearance of [something] in the subsequent attached figures... Figure 5 Similar textual descriptions can be found here.

[0149] It should be understood that, regarding Figure 5 The screenshot 500 and page analysis result 510 shown can be displayed on the screen during actual use of the mobile phone, or the mobile phone may not display the screenshot 500 and page analysis result 510 on the screen. This application embodiment does not impose any limitations on this.

[0150] S403. The Any Door obtains the page analysis results and matches them with the selected range to obtain the transmission object.

[0151] As one possible implementation, the AnyDoor can determine the selection range based on the trajectory data included in the knuckle events. Next, the AnyDoor matches the selection range with the layout analysis results.

[0152] As one possible example, any gate can use the smallest bounding rectangle of the trajectory data as the selection range.

[0153] It should be understood that in other examples, the aforementioned minimum bounding rectangle can also be a bounding ellipse, a bounding circle, a bounding regular polygon, a minimum inscribed rectangle, a minimum inscribed ellipse, etc., and can be designed according to actual usage needs. In the embodiments of this application, the minimum bounding rectangle whose selection range is trajectory data is used as an example for introduction, and the embodiments of this application do not impose any limitations on this.

[0154] In some embodiments, the matching results may include text or images.

[0155] As a possible example, AnyDoor can determine the valid text region and the valid image region based on the selected area and the page analysis results. Next, AnyDoor determines whether the matching result is text or an image based on the area of ​​the valid text region and the area of ​​the valid image region.

[0156] For example, the page analysis result includes at least one image region and at least one text region; if the ratio of a region within the selected range that belongs to the target text region to the target text region is greater than or equal to a first preset ratio, the arbitrary gate determines that the region within the selected range that belongs to the target text region is a valid text region. Then, the arbitrary gate determines that the region within the selected range other than the valid text region is a valid image region. The target text region belongs to at least one of the aforementioned text regions.

[0157] Understandably, the target text region is the text region that intersects with the selected area. The first preset ratio can be 60%, 65%, 80%, etc. Specifically, the value of the first preset ratio can be set according to actual usage needs, and this embodiment does not limit this.

[0158] For example, if the area of ​​the effective text region is larger than the area of ​​the effective image region, the arbitrary gate determines that the transmission object is text; if the area of ​​the effective image region is larger than the area of ​​the effective text region, the arbitrary gate determines that the transmission object is an image.

[0159] As a possible example, if the transmitted object is text, AnyDoor can select the text within a range in a screenshot as the transmitted object; if the transmitted object is an image, AnyDoor can select the image within a range in a screenshot as the transmitted object.

[0160] Optionally, the arbitrary gate can also expand the selection range to obtain and obtain the transmission object based on the expanded selection range.

[0161] For example, if the arbitrary gate determines that the transmission object is an image, and the ratio of the area of ​​the region included in the effective image region and belonging to the target image region to the area of ​​the target image region is less than a second preset ratio, the arbitrary gate will select the image within the selected range as the transmission object. The second preset ratio can be 60%, 75%, 80%, etc. Specifically, the value of the second preset ratio can be set according to actual usage needs, and this application embodiment does not limit this.

[0162] For example, if the arbitrary gate determines that the transmission object is an image, and the ratio of the area of ​​the region included in the valid image region and belonging to the target image region to the area of ​​the target image region is greater than or equal to a second preset ratio, but less than 100%, the arbitrary gate will use the image included in the selected range after merging with the target image region as the matching result. In other words, if the selected range includes most (e.g., the second preset ratio) of the target image region, the arbitrary gate expands the selected range so that the transmission object obtained by the arbitrary gate includes the complete (all) target image region. It can be understood that the target image region is the image region that intersects with the selected range.

[0163] For example, if the arbitrary gate determines that the matching result is text, and the ratio of the area of ​​the region that is included in the valid text region and belongs to the target text region to the area of ​​the target text region is less than 100%, the arbitrary gate will use the text included in the selected range after merging with the target text region as the transmission object. In other words, if the selected range includes most of the target text region, the arbitrary gate expands the selected range so that the transmission object obtained by the arbitrary gate includes the complete (all) target image region.

[0164] As a possible example, the "anywhere door" could carry the aforementioned transmitted object using a hash map. This hash map could include clip data. If the transmitted object is an image, the clip data includes a flag indicating the selected screenshot, such as a Uniform Resource Locator (URL). This selected screenshot represents the images included in the selection range; alternatively, it could represent the images included in the expanded selection range. If the transmitted object is text, the clip data includes the text itself, encompassing the target text region that has been included in the selection range.

[0165] For example, see Figure 6 Part A of the text Figure 6 Part A of the diagram shows page analysis result 600, which includes image region 602. Next, the arbitrary gate obtains selection range 601. Then, based on the fact that the area of ​​the valid image region is greater than the area of ​​the valid text region, the arbitrary gate determines that the transmission object is an image. Next, based on the fact that the area of ​​the region included in selection range 601 and belonging to image region 602 is more than 60% of the area of ​​image region 602, the arbitrary gate adjusts the size of the selection range; for example, it merges image region 602 with selection range 601 to obtain selection range 603, and uses the image included in selection range 603 as the transmission object. As an example, the arbitrary gate can use hashmap0 to carry the above transmission object, where hashmap0 includes the URL representing the image of the transmission object.

[0166] For example, see again Figure 6 Part B of the text Figure 6Part B of the diagram shows the page analysis result 610, which includes an image region 611 and a text region 613. Next, the arbitrary gate obtains a selection range 612. Then, based on the fact that the area of ​​the valid image region is greater than the area of ​​the valid text region, the arbitrary gate determines that the transmission object is an image. Next, based on the fact that the area of ​​the region included in the selection range 612 and belonging to the image region 611 does not exceed 60% of the area of ​​the image region 611, the arbitrary gate does not adjust the size of the selection range and uses the image included in the selection range 612 as the transmission object. As an example, the arbitrary gate can use a hash map1 to carry the above transmission object, where hash map1 includes the URLs representing the images of the transmission objects.

[0167] For example, see again Figure 6 Part C in Figure 6 Part C of the diagram shows page analysis result 620, which includes image region 621 and text region 623. Next, the arbitrary gate obtains a selection range 622. Then, based on the fact that the ratio of the area of ​​the text region 623 included in the selection range to the area of ​​the text region 623 is less than 60%, the arbitrary gate determines that the selection range 622 does not include a valid text region and includes a valid image region. Next, based on the fact that the area of ​​the valid image region is greater than the area of ​​the valid text region, the arbitrary gate determines that the transmission object is an image. Then, based on the fact that the proportion of the image region 621 occupied by the selection range 622 does not exceed 60%, the arbitrary gate does not adjust the size of the selection range and uses the image included in the selection range 622 as the transmission object. As an example, the arbitrary gate can use hash map2 to carry the above transmission object, where hash map2 includes the URL of the image representing the transmission object.

[0168] For example, see Figure 7 Part A of the text Figure 7 Part A of the diagram shows page analysis result 700, which includes text region 702. Next, the arbitrary gate obtains selection range 701. Then, based on the fact that the area of ​​the effective text region is greater than the area of ​​the effective image region, the arbitrary gate determines that the transmission object includes text. Next, based on the fact that the area of ​​the region included in selection range 701 and belonging to the line text region 702 is more than 60% of the area of ​​the line text region 702, the arbitrary gate adjusts the size of the selection range. For example, the arbitrary gate merges the line text region 702 with selection range 701 to obtain selection range 703. Then, the arbitrary gate uses the text included in selection range 703 as the transmission object. As an example, the arbitrary gate can use hash map3 to carry the above transmission object, where hash map3 includes the text representing the included transmission object, such as "AAAAAAABBAAAAAAAA".

[0169] For example, see Figure 7 Part B of the text Figure 7 Part B of the diagram shows the page analysis result 710, which includes text regions 711, 712, and image regions 715. Next, the arbitrary gate obtains a selection range 713. Then, based on the fact that the area of ​​the effective text region is greater than the area of ​​the effective image region, the arbitrary gate determines that the transmission object includes text. Then, based on the fact that the area of ​​the region included in selection range 713 and belonging to text region 712 is greater than 60% of the area of ​​text region 712, the arbitrary gate merges selection range 713 with text region 712. And, based on the fact that the area of ​​the region included in selection range 713 and belonging to text region 711 is greater than 60% of the area of ​​text region 711, the arbitrary gate merges selection range 713 with text region 711. The arbitrary gate obtains an adjusted selection range 714. Then, the arbitrary gate uses the text included in selection range 714 as the transmission object. As an example, any door can carry the aforementioned transmission object through hash map4, which includes the text representing the transmission object, such as "AAAAAAABBAAAAAAAA" or "AAAAAAAAACCAAAAAA".

[0170] It should be pointed out that, Figure 6 , Figure 7 The content shown is for illustrative purposes only. In actual use of the phone, the screen may display the above information. Figure 6 , Figure 7 The content shown, or the phone's screen, may not be displayed. Figure 6 , Figure 7 The contents shown in this application embodiment are not intended to limit anything.

[0171] In some embodiments, step S403 may include: an arbitrary gate acquiring at least one target text region that intersects with the selected range within a text region. Next, the arbitrary gate acquires the ratio of the intersection of the selected range and the target text region to the target text region. If this ratio is greater than or equal to the first threshold, the arbitrary gate acquires the remaining portion of the selected range excluding the intersection. Then, the arbitrary gate determines the transmission object based on the area of ​​the intersection and the areas of the other portions.

[0172] The arbitrary gate determines the transmission object based on the area of ​​the intersection region and the area of ​​other regions. This can include: if the area of ​​the intersection region is larger than the area of ​​other regions, the arbitrary gate merges the selected range with the target text region to obtain a second merged region. Next, the arbitrary gate determines the text included in the second merged region as the transmission object. Alternatively, if the area of ​​other regions is larger than the area of ​​the intersection region, the arbitrary gate determines that the transmission object is the image included in the selected range.

[0173] In some implementations, when the area of ​​other portions is larger than the area of ​​the intersection portion, the electronic device acquires at least one target image region that intersects with the selection range within an image region. Then, if the ratio of the intersection region of the selection range and the target image region to the target image region is greater than or equal to a second threshold, the electronic device merges the target image region and the selection range to obtain a first merged region. The electronic device then determines that the transmission target is the image included in the first merged region. Alternatively, if the ratio of the intersection region of the selection range and the target image region to the target image region is less than the second threshold, the electronic device determines that the transmission target is the image included in the selection range.

[0174] In some embodiments, step S403 may include: an arbitrary gate acquiring at least one target text region within a text region that intersects with the selected range. Next, the arbitrary gate acquires the ratio of the intersection of the selected range and the target text region to the target text region. If this ratio is less than the first threshold, the arbitrary gate determines that the image included in the selected range is the transmission object.

[0175] The process of determining that the image included in the selection range is the transmission object can include: the electronic device acquiring at least one target image region that intersects with the selection range within an image region. Then, if the ratio of the intersection region of the selection range and the target image region to the target image region is greater than or equal to a second threshold, the electronic device merges the target image region and the selection range to obtain a first merged region. The electronic device then determines that the transmission object is the image included in the first merged region. Alternatively, if the ratio of the intersection region of the selection range and the target image region to the target image region is less than the second threshold, the electronic device determines that the transmission object is the image included in the selection range.

[0176] S404. The intent frame obtains the entity based on the transmitted object.

[0177] Entities can include text entities or image entities.

[0178] As one possible implementation, when the transmitted object is an image, the intent framework can perform image recognition on the image to obtain image entities.

[0179] For example, when the transmitted object includes a flag for a selected screenshot, the intent framework performs image recognition on the selected screenshot using a CV model to obtain the image entity.

[0180] As one possible implementation, when the transmitted object is text, the intent framework can perform natural language processing on the text to obtain text entities.

[0181] For example, when the transmitted object includes text, the intent framework can perform natural language processing on the text using an NLP model to obtain text entities.

[0182] S405. The service management framework obtains a list of services corresponding to an entity based on that entity.

[0183] The service management framework may include the aforementioned service loading module.

[0184] As a possible example, the service management module can obtain user intent based on entities. Then, the service management framework can match the user intent to obtain a list of services. Furthermore, the service management framework can add user-specified services to this list. Details regarding user-specified services are provided below and will not be elaborated upon here.

[0185] As one possible example, the service management module can access multiple services locally on the mobile phone, or it can access multiple services from a cloud service that has established a communication connection with the mobile phone. Each service includes a service number, such as a service ID. The service management framework can have a pre-defined mapping table between entity and service IDs, which stores a list of service IDs corresponding to each entity. The list of service IDs corresponding to each entity includes the IDs of at least one service associated with that entity. In other words, the service management framework can obtain the list of services corresponding to an entity based on the aforementioned mapping table.

[0186] S406. The intent framework sorts the services included in the service list.

[0187] As one possible example, in step S406, the intent framework can sort the services included in the service list according to the frequency of service usage. Alternatively, it can sort them according to the service names. This embodiment of the application does not limit the sorting method of the intent framework.

[0188] In some embodiments, the intent framework can sort services according to the priority of the services included in the service list. The service priority can be user-indicated; for example, a user can indicate a higher priority for a service through the service list interface. A description of the service list interface is provided below and will not be repeated here. Alternatively, the service priority can also be related to the frequency with which the service is used by the user. For example, a service with high user frequency has a higher priority, and vice versa. The statistics on the frequency of service triggering can be performed by the intent framework or by other software modules of the mobile phone, such as the service management framework, service frequency statistics module, etc. This application embodiment does not limit this approach.

[0189] In another implementation, the intent framework may not sort the services included in the service list and may directly send the service list to any gate. That is, in some implementations, step S406 is optional.

[0190] S407. Any Door displays the service trigger interface based on the sorted service list.

[0191] As one possible implementation, AnyDoor can use the aforementioned screenshot as the background, the service icons included in the service list as the foreground, and the text or selected screenshot of the transmission object as the foreground to draw the service trigger interface. Then, the phone's display shows the service trigger interface.

[0192] As one possible implementation, AnyDoor can display icons of the services included in the service list at a specified location on the service triggering interface, based on the sorted service list.

[0193] The specified position can be preset or obtained based on the knuckle selection operation. Further details regarding the specified position will be provided below and will not be repeated here.

[0194] For example, see Figure 8 The AnyDoor displays the service trigger interface 800 based on the sorted service list. The service trigger interface 800 includes icons for multiple services. These services are all part of the service list. In other words, the AnyDoor can display the icons of the services included in the service list on the service trigger interface.

[0195] For another example, the portal can determine the relative position of a service icon to a specified location based on the order of services in the sorted service list. For instance, the order of services in the service list might be: Gallery Service, Shopping Service, WeChat. TM Services, Moments TMServices include information services and note-taking services. The order of these services is from most important to least important; for example, image gallery services are more important than shopping services. See also... Figure 8 The designated area 801 of the service trigger interface 800 includes icons for multiple services, such as the Gallery service icon 801A, the Shopping service icon 801B, the WeChat service icon 801C, the Moments service icon 801D, the Message service icon 801E, and the Notes service icon 801F. Among these, the Gallery service icon is displayed at the top of the designated area 801, with the Gallery service icon 801A appearing first. Next, the Shopping service icon is displayed below the Gallery service icon, with the Shopping service icon 801B appearing below the Gallery service icon 801A. Similarly, the process of displaying other service icons in the designated area 801 is similar to that of the Gallery and Shopping service icons, and will not be elaborated further.

[0196] It should be understood that the relative position of the service icon to the specified location can be determined according to the order of services in the sorted service list. Many other designs are also possible, and this application embodiment does not limit this.

[0197] Optional, see again Figure 8 The service trigger interface 800 may also include one or more of the following: a selection box 802, a text-image conversion control 803, an editing control 804, and a save control 805.

[0198] Selection box 802 is used to display the selection range of the knuckle selection operation triggered by the user. Among them, in Figure 8 The content included in selection box 802 can be highlighted. For example, Figure 8 The selection box 802 may contain highlighted content (e.g., without grayscale display), and / or other elements included in the service trigger interface 800 may be highlighted in grayscale (e.g., with grayscale display). These other elements can be understood as all elements included in the service trigger interface 800 other than controls and service icons.

[0199] Furthermore, the selection box 802 also provides a means for the user to adjust the selection range. For example, the user can drag the selection box 802 to adjust the selection range by changing its size or position. It should be understood that after the user adjusts the selection range, the phone can repeat the steps S403-S407. For instance, the phone updates the service list and adjusts the service icons displayed on the service triggering interface according to the updated service list.

[0200] The image-to-text conversion control 803 is used to provide a user-triggered text recognition function for content within a selected range. For example, a user can click on the image-to-text conversion control 803. In response to the trigger operation of the image-to-text conversion control 803, the arbitrary door updates the transmission object, and the mobile phone executes the above steps S404-S407 according to the updated transmission object to display the service trigger interface.

[0201] For example, when the transmitted object is text, in response to the triggering operation of the image-text conversion control 803, the arbitrary door updates the transmitted object to an image; when the transmitted object is an image, in response to the triggering operation of the image-text conversion control 803, the arbitrary door updates the transmitted object to text.

[0202] In some embodiments, if the image included in selection box 802 does not include text, in response to a triggering operation on the image-to-text conversion control, the mobile phone displays a prompt indicating that the image included in selection box 802 does not include text. It is understood that the image included in selection box 802 is... Figure 8 In the corresponding scenario, the mobile phone executes the transmission object obtained in step S403.

[0203] For example, see Figure 8 The mobile phone displays a service trigger interface 800, which includes an image-to-text conversion control 803. Next, in response to a trigger operation on the image-to-text conversion control 803, the mobile phone displays a service trigger interface 810. The service trigger interface 810 includes a prompt 811. Prompt 811 indicates that the image included in the selection box 802 does not include text. For example, prompt 811 could be "The currently selected area does not include text."

[0204] The editing control 804 is used to provide a user-triggered image editing interface on the mobile phone. The image editing interface may include the content of the selection box 802. Alternatively, the image editing interface may include a screenshot taken by the mobile phone when it receives a knuckle selection operation.

[0205] The save control 805 is used to provide a user-triggered save operation on the content included in the selection box 802 via the mobile phone. For example, in response to the trigger operation of the save control 805, the device saves the content included in the selection box 802. For instance, the device stores the content included in the selection box 802 to a preset storage location.

[0206] It should be understood that the process corresponding to steps S400-S407 above can also be called the opening process. That is, in response to the knuckle selection operation, the mobile phone executes the opening process. The purpose of the mobile phone executing the opening process is to enable the flow of content selected by the user between different applications. For example, content selected by the user through the knuckle selection operation on the web page interface flows to the gallery application, shopping application, WeChat application, etc. Therefore, in some embodiments, the above-mentioned service trigger interface can also have more other designs. For example, the service icon can be displayed in the form of a floating window, or in the form of a floating capsule, etc.; specifically, it can be designed according to actual usage needs, and the embodiments of this application are not limited in this regard.

[0207] Optionally, the information transmission method provided in this application embodiment may further include step S408 after step S407.

[0208] S408. In response to a trigger operation on the target service, AnyDoor starts the target service.

[0209] It should be understood that once the target service is started, it can perform preset functions based on the content included in the selection. For example, a gallery service can provide the function of saving the content included in the selection to the gallery. A shopping service can provide the function of searching for the content included in the selection within shopping software, and so on.

[0210] The aforementioned triggering operations on the target service can take many forms.

[0211] For example, see Figure 9 The phone displays a service trigger interface 900, which includes a selection box 902 and a WeChat service icon 901C. Next, in response to the phone receiving a user's action of moving the WeChat service icon from the selection box 902 to 901C, the app launches the WeChat service, and the WeChat service displays an image sending interface 920. This user action of moving the content from the selection box 902 to the WeChat service icon 901C can also be described as dragging the content of the selection box 902 to the WeChat service icon 901C.

[0212] For example, see again Figure 9 The phone displays a service trigger interface 910, which includes a selection box 912 and an icon 911D for the Moments service. Next, in response to the phone receiving the user's click on the Moments service icon 911D, the "AnyDoor" function launches the Moments service, which then displays the Moments sharing interface 930.

[0213] The technical solutions provided in the embodiments of this application will be further described below in conjunction with the user's experience of using the mobile phone.

[0214] Before introducing how users use their phones, let's first introduce how users activate the "Anywhere Door" function on their phones.

[0215] For example, see Figure 10 The phone displays desktop 1000, which includes an icon 1001 for the settings application. Next, in response to a triggering operation on the settings application icon 1001, the phone displays settings interface 1010; settings interface 1010 includes a smart assistant menu button 1011. Then, in response to a triggering operation on the smart assistant menu button 1011, the phone displays the smart assistant menu interface 1020; smart assistant menu interface 1020 includes an "Anywhere Door" menu button 1021. Finally, in response to a triggering operation on the "Anywhere Door" menu button 1021, the phone displays the "Anywhere Door" menu interface 1030.

[0216] It should be understood that the aforementioned triggering operation may include a click operation, a double-click operation, etc. The specific design can be tailored to actual usage requirements, and this application embodiment does not impose any restrictions on the specific operation form of the triggering operation. Further details regarding the triggering operation in this application embodiment can be found here.

[0217] The Any Door menu interface 1030 includes an Any Door switch 1031 and a knuckle switch 1032. The switches can be in an on or off state. The Any Door switch 1031 is used to turn the Any Door function of the phone on or off. For example, when the Any Door switch 1031 is on, the Any Door function of the phone is enabled. Similarly, when the Any Door switch 1031 is off, the Any Door function of the phone is disabled. The knuckle switch is used to turn the phone's function of performing the door-opening process in response to a knuckle selection operation on or off. When the knuckle switch 1032 is on, the phone can perform the door-opening process. When the knuckle switch 1032 is off, the phone does not perform the door-opening process. It can be understood that in some other embodiments, when the Any Door switch 1031 is on, the phone can perform the door-opening process in response to a long press operation on an icon or text.

[0218] As one possible implementation, the user can also indicate the priority of a service. Optionally, the AnyDoor menu interface 1030 also includes a custom service button 1033. In response to triggering the custom service button 1033, the phone displays a service list interface 1040. The service list interface 1040 includes a first service list 1042 and a second service list 1041. The first service list 1042 may include icons of multiple services, and the services included in the second service list 1041 have higher priority. The user can move the icon of a service included in the first service list 1042 to the second service list 1041 to indicate higher priority for that service. For example, the first service list 10402 includes an icon 1042A for the search service; then, in response to moving the icon 1042A of the search service from the first service list 1042 to the second service list 1041, the phone assigns higher priority to the search service.

[0219] In the technical solution provided in this application embodiment, the user can select an image through a selection operation, and the mobile phone displays the icon of the service matching the image on the service trigger interface.

[0220] For example, see Figure 11 The mobile phone displays a webpage display interface 1100, which includes a first image 1101. Next, in response to a knuckle selection operation 1103, the mobile phone displays a service trigger interface 1110; the service trigger interface 1110 includes a selection box 1112, which includes the first image 1111; and the service trigger interface 1110 also includes icons for multiple services that match the first image.

[0221] The aforementioned knuckle selection operation 1103 can be performed by a user moving their knuckle on the phone screen along a trajectory indicated by a dashed arrow. This knuckle selection operation 1103 selects a portion of the first image 1101, for example, selecting a first image larger than an image threshold. The image threshold can be 60%. It can be understood that the knuckle selection operation selecting a first image larger than the image threshold means that the bounding rectangle of the knuckle selection operation's trajectory includes the first image larger than the image threshold.

[0222] Optionally, in the web page display interface 1100, the content included in the selection box 1112 can be highlighted.

[0223] Optionally, the mobile phone can execute the above steps S400-S407 to display the service trigger interface 1110. During step S403, the "AnyDoor" determines that the selection range matches the first image based on the fact that the proportion of the selection range occupying the first image exceeds 60%. The selection range can be the smallest bounding rectangle of the trajectory of the knuckle selection operation 1103. Furthermore, the "AnyDoor" modifies the selection range to include the entire first image. For example, the "AnyDoor" expands the smallest bounding rectangle of the trajectory of the knuckle selection operation 1103 to include the entire first image.

[0224] In the technical solution provided in this application embodiment, the user can adjust the image on the service trigger interface, and the mobile phone displays the icon of the service corresponding to the adjusted image on the service trigger interface.

[0225] For example, see again Figure 11 After the phone displays the service trigger interface 1110, in response to a trigger operation on the selection box 1112, the phone displays the selection box adjustment interface 1120. The user can adjust the size and / or position of the selection box 1121 in the selection box adjustment interface 1120, so that the selection box includes different content, allowing selection of different content. Subsequently, in response to an adjustment operation on the selection box 1121, the phone displays the service trigger interface 1130; the service trigger interface 1130 includes the selection box 1131, which includes the first content of the first image; and the service trigger interface 1130 also includes icons of multiple services matching the first content. The first content may be "pet dog".

[0226] It should be noted that the order of the service icons in service trigger interfaces 1110 and 1130 is different. For example, the gallery service icon is at the top of service trigger interface 1110, while the shopping service icon is at the top of service trigger interface 1130. In other words, the phone can display service icons on the service trigger interface based on the content included in the selection box. When the content included in the selection box is different, the phone displays the service icons in different orders, or displays different icons altogether.

[0227] It should be understood that the icons at the top make it easier for users to trigger services. Therefore, by changing the display order of service icons or displaying different icons, mobile phones can make it easier for users to trigger the services corresponding to the selection box. This further enhances the flexibility of electronic devices in displaying services, improves the ease with which users can trigger services, and enhances the user experience.

[0228] In the technical solution provided in this application embodiment, the user can select an image through a selection operation, and the mobile phone displays the icon of the service matching the image on the service trigger interface.

[0229] For example, see Figure 12 The mobile phone displays a webpage display interface 1200, which includes a first image 1201. Next, in response to a knuckle selection operation 1203, the mobile phone displays a service trigger interface 1210; the service trigger interface 1210 includes a selection box 1212, which includes the first image 1211; and the service trigger interface 1210 also includes icons for multiple services that match the first image.

[0230] Specifically, the aforementioned knuckle selection operation 1203 can be described as the user moving their knuckle on the phone screen along a trajectory indicated by a dotted line with an arrow. This knuckle selection operation 1203 selects all of the first images 1201. It can be understood that the knuckle selection operation selecting all of the first images means that the bounding rectangle of the trajectory of the knuckle selection operation includes all of the first images.

[0231] Optionally, the mobile phone can perform the above steps S400-S407 to display the service trigger interface 1210.

[0232] For example, see again Figure 12 The mobile phone displays a webpage display interface 1220, which includes a first image 1221. Next, in response to a knuckle selection operation 1223, the mobile phone displays a service trigger interface 1230; the service trigger interface 1230 includes a selection box 1232, which includes the first content of the first image; and the service trigger interface 1230 also includes icons for multiple services matching the first content. The first content could be "pet dog".

[0233] Specifically, the aforementioned knuckle selection operation 1223 can be a user moving their knuckle on the phone screen along a trajectory indicated by a dotted line with an arrow. This knuckle selection operation 1223 selects the first content of the first image 1221. It can be understood that the knuckle selection operation selects the first content of the first icon, meaning that the bounding rectangle of the knuckle selection operation's trajectory includes the first content.

[0234] Optionally, the mobile phone can perform the above steps S400-S407 to display the service trigger interface 1230. During step S403, the arbitrary door determines that the selected area matches the content it includes, i.e., the selected area matches the first content, based on the fact that the proportion of the selected area occupying the first image does not exceed 60%.

[0235] It should be clarified that, in comparison with the above Figure 11 and the above Figure 12 As can be seen, when the user's knuckle operation selects most of the first image, the phone can expand the selection range of the knuckle selection operation, for example, to include all of the first images. Therefore, the phone can display icons for services matching the first image. For example, based on selection box 1112 including the first image 1111, the phone can display icons for services matching the first image on service trigger interface 1110; based on selection box 1212 including the first image 1211, the phone can display icons for services matching the first image on service trigger interface 1210. The icons for services included in service trigger interface 1110 and service trigger interface 1210 can be the same.

[0236] In the technical solution provided in this application embodiment, the user can select text through a selection operation, and the mobile phone displays the icon of the service matching the text on the service trigger interface.

[0237] For example, see Figure 13 The mobile phone displays a webpage display interface 1300, which includes multiple lines of text. Next, in response to a knuckle selection operation 1302, the mobile phone displays a service trigger interface 1310; the service trigger interface 1310 includes a selection box 1311, which includes multiple lines of text; and the service trigger interface 1310 also includes icons for multiple services that match the text. The aforementioned multiple lines of text can be... Figure 13 The characters “AAAAAAABBAAAAAAAA”, “AAAAAAAAACCAAAAAA”, and “AAAAAAAAAAAAAAAAA” are listed.

[0238] The aforementioned knuckle selection operation 1302 can be performed by a user moving their knuckle on the phone screen along a trajectory indicated by a dotted arrow. This knuckle selection operation 1302 selects a portion of the multi-line text, for example, selecting multiple lines of text exceeding a certain text threshold. The text threshold can be 65%. It can be understood that the knuckle selection operation selecting multiple lines of text exceeding the text threshold means that the bounding rectangle of the knuckle selection operation's trajectory includes the multiple lines of text exceeding the text threshold.

[0239] Optionally, in the web page display interface 1310, the content included in the selection box 1311 can be highlighted.

[0240] Optionally, the mobile phone can execute the above steps S400-S407 to display the service trigger interface 1310. During step S403, the "AnyDoor" determines that the selection range matches the multi-line text based on the fact that the proportion of the selected area occupying multiple lines of text exceeds 60%. Furthermore, the "AnyDoor" modifies the selection range to include all the multi-line text. For example, the "AnyDoor" expands the minimum bounding rectangle of the trajectory of the knuckle selection operation 1302, expanding the minimum bounding rectangle to include all the multi-line text.

[0241] Furthermore, in the technical solution provided in this application embodiment, the user can adjust the text on the service trigger interface, and the mobile phone displays the icon of the service corresponding to the adjusted text on the service trigger interface.

[0242] For example, see again Figure 13 The phone displays a service trigger interface 1310. Service trigger interface 1310 also includes a text selection control 1312. Next, after the phone displays service trigger interface 1310, in response to a trigger operation on the text selection control 1312, the phone displays a text adjustment interface 1320. The user can adjust the text content selected by the text selection control 1322 in the text adjustment interface 1320. Then, in response to an adjustment operation on the text selection control 1322, the phone displays a service trigger interface 1330; service trigger interface 1330 includes a text selection control 1332, and the text content selected by the text selection control 1332 is "CC". Furthermore, service trigger interface 1330 also includes icons for multiple services matching "CC". Here, "CC" refers to a park.

[0243] Understandably, the text selected by the text selection control can be inverted. For example, if the unselected text is white on a black background, then the text selected by the text selection control can be black on a white background.

[0244] It should be noted that the service icons included in service trigger interface 1310 and service trigger interface 1330 are different. For example, service trigger interface 1310 includes a WeChat service icon at the top of service trigger interface 1310, while service trigger interface 1330 includes a travel service icon at the top of service trigger interface 1330. That is, the mobile phone can display the service icon on the service trigger interface based on the text selected by the text selection control.

[0245] In the technical solution provided in this application embodiment, the user can select a large number of texts through a selection operation, and the mobile phone displays the icon of the service matching the text on the service trigger interface.

[0246] For example, see Figure 14The mobile phone displays a webpage interface 1400, which includes multiple lines of text. Next, in response to a knuckle selection operation 1402, the mobile phone displays a service trigger interface 1410; the service trigger interface 1410 includes a text selection control 1412, which includes multiple lines of text, meaning the user has selected multiple lines of text using the text selection control. Furthermore, the service trigger interface 1410 also includes an icon for a service that matches the multiple lines of text. The aforementioned multiple lines of text can be... Figure 14 The characters “AAAAAAABBAAAAAAAA”, “AAAAAAAAACCAAAAAA”, and “AAAAAAAAAAAAAAAAA” are listed.

[0247] Specifically, the aforementioned knuckle selection operation 1402 can be performed by a user moving their knuckle on the phone screen along a trajectory indicated by a dotted line with an arrow. This knuckle selection operation 1402 selects all lines of text. It can be understood that the knuckle selection operation selects all lines of text, meaning that the smallest bounding rectangle of the trajectory of the knuckle selection operation includes all lines of text.

[0248] Optionally, the mobile phone can perform the above steps S400-S407 to display the service trigger interface 1410.

[0249] In the technical solution provided in this application embodiment, the user can select a small amount of text, and the mobile phone will display a service icon matching the image containing the aforementioned small amount of text on the service trigger interface. Then, the user can use the image-text conversion control to trigger the mobile phone to display the icon of the service matching the text.

[0250] For example, see Figure 15 The mobile phone displays a webpage interface 1500, which includes multiple lines of text. Next, in response to a knuckle selection operation 1502, the mobile phone displays a service trigger interface 1510; the service trigger interface 1510 includes a selection box 1512 and an image-text conversion control 1513. Furthermore, the service trigger interface 1510 also includes icons for multiple services that match the second content. The second content can be the content included in the selection box 1512, such as an image containing the text "AACCAA".

[0251] Specifically, the aforementioned knuckle selection operation 1502 can be performed by a user moving their knuckle on the phone screen along a trajectory indicated by a dotted arrow. This knuckle selection operation 1502 selects an image containing the text "AACCAA".

[0252] Optionally, the mobile phone can perform the above steps S400-S407 to display the service trigger interface 1510. During the execution of step S403, the "AnyDoor" modifies the selection range to match the image based on the fact that the proportion of the target text occupied by the selection range does not exceed the text threshold, and the "AnyDoor" determines that the selection range matches the content included in the selection range. The selection range can be the smallest bounding rectangle of the trajectory of the knuckle selection operation 1502, the target text is the text in the line containing "AACCAA", such as "AAAAAAAAACCAAAAAA"; the content included in the selection range can be the content included in the selection box 1512, such as an image containing the text "AACCAA".

[0253] Next, in response to the trigger operation of the image-text conversion control 1513, the mobile phone displays a service trigger interface 1520. The service trigger interface 1520 includes a selection box 1521, an image-text conversion control 1523, and a text selection control 1524. Furthermore, the service trigger interface 1520 may also include an icon for a service that matches the content selected by the text selection control 1524. For example, an icon for a service matching "AAAAAAAAACCAAAAAA".

[0254] Optionally, in response to the triggering operation of the image-text conversion control 1513, the mobile phone displays the service triggering interface 1520, and the selection range is adjusted. For example, the selection range is adjusted to match the target text, and the size of the selection range is adjusted so that the selection range includes all the target text. Here, the target text is the text in the line containing "AACCAA", such as "AAAAAAAAACCAAAAAA".

[0255] Optionally, the image-text conversion control 1523 included in the service trigger interface 1520 and the image-text conversion control 1513 included in the service trigger interface 1510 can have different states. For example, the image-text conversion control 1523 can be highlighted, or it can be de-highlighted. In other words, the mobile phone can indicate that the image-text conversion control has been triggered by highlighting the image-text conversion control 1523, and it can indicate that the image-text conversion control has not been triggered by de-highlighting the image-text conversion control 1513.

[0256] Subsequently, in response to the adjustment operation of the text selection control 1524, the mobile phone displays a service trigger interface 1530. The service trigger interface 1530 includes a text selection control 1534. The text selection control 1534 selects the text "CC". The service trigger interface 1530 also includes an icon for a service that matches the text "CC".

[0257] In the technical solution provided in this application embodiment, the user can select text and images through a selection operation. When there are more images than text, the mobile phone displays the icon of the service that matches the image on the service trigger interface.

[0258] For example, see Figure 16 The mobile phone displays a webpage display interface 1600, which includes a first image and multiple lines of text. Next, in response to a knuckle selection operation 1602, the mobile phone displays a service trigger interface 1610; the service trigger interface 1610 includes a selection box 1612. Furthermore, the service trigger interface 1610 also includes icons for multiple services that match the third content. The third content can be the content included in the selection box 1612, such as a portion of the first image, and the text "AAAAAAABBAAAAAAAA".

[0259] Specifically, the aforementioned knuckle selection operation 1602 can be performed by a user moving their knuckle on the phone screen along a trajectory indicated by a dotted line with an arrow. This knuckle selection operation 1602 selects a portion of the first image and the text "AAAAAAABBAAAAAAAA".

[0260] Optionally, the mobile phone can perform the above steps S400-S407 to display the service trigger interface 1610. During step S403, the arbitrary door establishes a matching relationship between the selection range and the text region based on the fact that the area of ​​the image region included in the selection range is greater than the area of ​​the text region.

[0261] The service trigger interface 1610 also includes a non-highlighted text-to-image conversion control 1611. After the mobile phone displays the service trigger interface 1610, in response to a trigger operation on the text-to-image conversion control 1611, the mobile phone displays the service trigger interface 1620. The service trigger interface includes the highlighted text-to-image conversion control 1621 and a text selection control 1624. Furthermore, the service trigger interface 1610 also includes an icon for the service that matches the content selected by the text selection control 1624.

[0262] In the technical solution provided in this application embodiment, the user can select text and images through a selection operation. When there is more text than images, the mobile phone displays the icon of the service that matches the text on the service trigger interface.

[0263] For example, see Figure 17The mobile phone displays a webpage interface 1700, which includes a first image and multiple lines of text. Next, in response to a knuckle selection operation 1702, the mobile phone displays a service trigger interface 1710; the service trigger interface 1710 includes a text selection control 1714. Furthermore, the service trigger interface 1710 also includes icons for multiple services that match the content selected by the text selection control.

[0264] Specifically, the aforementioned knuckle selection operation 1702 can be performed by a user moving their knuckle on the phone screen along a trajectory indicated by a dotted line with an arrow. This knuckle selection operation 1702 selects a portion of the first image, as well as the text “AAAAAAABBAAAAAAAA”, “AAAAAAAAACCAAAAAA”, and “AAAAAAAAAAAAAAAAA”.

[0265] Optionally, the mobile phone can perform the above steps S400-S407 to display the service trigger interface 1710. During step S403, the arbitrary door establishes a matching relationship between the selection range and the text region based on the fact that the area of ​​the text region included in the selection range is greater than the area of ​​the image region.

[0266] The service trigger interface 1710 also includes a highlighted image-to-text conversion control 1711. After the mobile phone displays the service trigger interface 1710, in response to a trigger operation on the image-to-text conversion control 1711, the mobile phone displays the service trigger interface 1720. The service trigger interface includes a non-highlighted image-to-text conversion control 1721 and a selection box 1722. The service trigger interface 1720 also includes an icon for a service that matches the fourth content. The fourth content can be the content included in the selection box 1722, such as a portion of the first image, and the text "AAAAAAABBAAAAAAAA", "AAAAAAAAACCAAAAAA", and "AAAAAAAAAAAAAAAAA".

[0267] It should be noted that, as seen from steps S402 and S403 above, the mobile phone's visual platform performs page analysis on the screenshot to obtain the page analysis results, and then matches the page analysis results with the knuckle event. Therefore, in the information transmission method provided in this application embodiment, the mobile phone can obtain the content selected by the user-triggered selection operation through a screenshot. Thus, for text included in videos or images, the mobile phone can obtain this text through a screenshot and display service icons based on this text.

[0268] For example, see Figure 18The phone displays a video playback interface 1800. The video playback interface 1800 includes a first video 1801, the video content of which includes first text, such as “AAAAACCAAAAA”. Next, in response to a knuckle selection operation 1802, the phone displays a service trigger interface 1810; the service trigger interface 1810 includes a text selection control 1814. Furthermore, the service trigger interface 1810 also includes an icon for a service that matches the content selected by the text selection control.

[0269] Specifically, the aforementioned knuckle selection operation 1802 can be performed by a user moving their knuckle on the phone screen along a trajectory indicated by a dotted line with an arrow. This knuckle selection operation 1802 selects the first text included in the first video 1801.

[0270] Optionally, the mobile phone can perform the above steps S400-S407 to display the service trigger interface 1810. During step S403, the arbitrary door establishes a matching relationship between the selection range and the text region based on the fact that the area of ​​the text region included in the selection range is greater than the area of ​​the image region.

[0271] In some implementations, the service icons included in the service triggering interface can have different states. For example, a service icon can be grayed out or not grayed out. A grayed-out state indicates that the service is unavailable to the user. For instance, if a service's input is limited to text, then if the selection range includes an image—that is, if the selection range matches an image area—the service icon can be grayed out. Similarly, if a service's input is limited to images, then if the selection range includes text—that is, if the selection range matches a text area—the service icon can be grayed out.

[0272] It is understood that the grayed-out and ungrayed-out states described above are merely examples. In actual use, the mobile phone can also use other different states to indicate whether the service can be triggered by the user. Specifically, the design can be tailored to actual usage needs, and this application embodiment does not limit this.

[0273] For example, see Figure 19The mobile phone displays a webpage display interface 1900, which includes a first image 1901. Next, in response to a knuckle selection operation 1903, the mobile phone displays a service trigger interface 1910; the service trigger interface 1910 includes a selection box 1912, which includes the first content of the first image; and the service trigger interface 1910 also includes icons of multiple services matching the first content. The first content may be "pet dog". The search service icon 1911 included in the service trigger interface 1910 may be grayed out. Understandably, based on the fact that the search service input is text and the first content included in the selection box 1912 is an image, the mobile phone displays the search service icon 1911 in gray. The search service may be a user-specified service with high priority. The process of the user indicating a high-priority search service can be found above. Figure 10 The corresponding description.

[0274] In some embodiments, the phone can perform the door-opening process from different opening directions. That is, the designated location of the service-carrying icon can be related to the location where the selection operation is triggered.

[0275] For example, the specified location can be a first preset location or a second preset location, where the first preset location is located on the left half of the screen and the second preset location is located on the right half of the screen. The mobile phone can determine whether the specified location is the first preset location or the second preset location based on the distance between the knuckle operation and the screen edge.

[0276] For example, see Figure 20 The phone displays a webpage interface 2000. Next, in response to a knuckle selection operation 2003, the phone displays a service trigger interface 2010. The service trigger interface includes a service icon. The knuckle selection operation 2003 can be performed by the user moving their knuckle across the phone screen along a trajectory indicated by a dotted arrow. During the display of the service trigger interface 2010, based on the distance 2001 between the knuckle selection operation 2003 and the left edge of the screen being less than the distance 2002 between the knuckle selection operation 2003 and the right edge of the screen, the phone places the service icon at a first preset position. That is, based on the distance 2001 being less than the distance 2002, the phone displays the service icon on the left side of the screen.

[0277] For example, see again Figure 20The phone displays a webpage interface 2020. Next, in response to a knuckle selection operation 2023, the phone displays a service trigger interface 2030. The service trigger interface includes a service icon. The knuckle selection operation 2023 can be performed by the user moving their knuckle across the phone screen along a trajectory indicated by a dotted arrow. During the display of the service trigger interface 2030, based on the distance 2022 between the knuckle selection operation 2023 and the right edge of the screen being less than the distance 2021 between the knuckle selection operation 2023 and the left edge of the screen, the phone places the service icon at a second preset position. That is, based on the distance 2021 being less than the distance 2022, the phone displays the service icon on the right side of the screen.

[0278] It should be noted that the personal information used in the technical solution of this application is limited to information for which individual consent has been obtained, including but not limited to notifying and reminding users to read the relevant user agreement (notification) and sign the agreement (authorization) which includes authorization of relevant user information before users use the function.

[0279] Based on the algorithmic steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementations should not be considered beyond the scope of this application.

[0280] This embodiment can divide the electronic device into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0281] This application also provides an electronic device, such as... Figure 21 As shown, the electronic device may include one or more processors 2101, memory 2102 and communication interfaces 2103.

[0282] The memory 2102, communication interface 2103, and processor 2101 are coupled together. For example, the memory 2102, communication interface 2103, and processor 2101 can be coupled together via bus 2104.

[0283] The communication interface 2103 is used for data transmission with other devices. The memory 2102 stores computer program code. The computer program code includes computer instructions, which, when executed by the processor 2101, cause the electronic device to perform the relevant method steps in the above-described method embodiments of this application.

[0284] The processor 2101 can be a processor or controller, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in connection with this disclosure. The processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0285] Bus 2104 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 2104 can be categorized into address bus, data bus, control bus, etc. For ease of representation, Figure 21 The symbol is represented by only one line, but this does not mean that there is only one bus or one type of bus.

[0286] This application also provides a chip system, such as... Figure 22 As shown, the chip system 2200 includes at least one processor 2201 and at least one interface circuit 2202. The processor 2201 and the interface circuit 2202 are interconnected via lines. For example, the interface circuit 2202 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 2202 can be used to send signals to other devices (e.g., the processor 2201). Exemplarily, the interface circuit 2202 can read instructions stored in the memory and send those instructions to the processor 2201. When the instructions are executed by the processor 2201, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application embodiment does not specifically limit this.

[0287] This application also provides a computer-readable storage medium storing computer program code. When the processor executes the computer program code, the electronic device executes the relevant method steps in the above method embodiments.

[0288] This application also provides a computer program product that, when run on a computer, causes the computer to execute the relevant method steps described in the above method embodiments.

[0289] The electronic devices, computer-readable storage media, or computer program products provided in this application are all used to perform the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0290] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0291] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0292] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0293] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0294] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0295] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for transmitting information, characterized in that, The method is applied to an electronic device, and the method includes: Display the first screen; Receive a first selection operation applied to the first interface, wherein the first selection operation is a selection operation; In response to the first selection operation, the selection trajectory corresponding to the first selection operation is displayed, and a screenshot of the first interface is obtained. Page analysis is performed on the screenshot to obtain page analysis results, which include text areas and / or image areas in the first interface; The transmission object is determined based on the selected trajectory and the page analysis results; After the transmission object is determined, a second interface is displayed. The second interface includes: the deformed screenshot, the transmission object marked on the deformed screenshot, and the service icon of the service corresponding to the transmission object.

2. The method according to claim 1, characterized in that, The step of determining the transmission object based on the selected trajectory and the page analysis results includes: The selection range is obtained based on the circled trajectory; the selection range represents the content selected by the first selection operation on the first interface. The transmission object is obtained by matching the selected range with the page analysis results.

3. The method according to claim 2, characterized in that, The page analysis result includes: at least one text region and at least one image region; matching the selected area and the page analysis result to obtain the transmission object includes: Obtain the target text region that intersects with the selected range in at least one text region; Obtain the ratio of the intersection of the selected range and the target text region to the target text region; If the ratio is greater than or equal to the first threshold, obtain the portion of the selection range other than the intersection portion; The transmission object is determined based on the area of ​​the intersection and the area of ​​the other parts.

4. The method according to claim 3, characterized in that, The step of determining the transmission object based on the area of ​​the intersection portion and the area of ​​the other portions includes: If the area of ​​the other portions is greater than the area of ​​the intersection portion, the transmission object is determined to be an image included in the selection range.

5. The method according to claim 4, characterized in that, Before determining that the transmission object is an image included in the selection range, the method further includes: Obtain the target image region in the at least one image region that intersects with the selected range; If the ratio of the intersection of the selected range and the target image region to the target image region is greater than or equal to a second threshold, the target image region and the selected range are merged to obtain a first merged region. The object to be transmitted is determined to be the image included in the first merged region; The determination that the transmission object is an image included in the selected range includes: If the ratio of the intersection of the selected range and the target image region to the target image region is less than a second threshold, the transmission object is determined to be an image included in the selected range.

6. The method according to any one of claims 3-5, characterized in that, The step of determining the transmission object based on the area of ​​the intersection portion and the area of ​​the other portions includes: If the area of ​​the intersection is greater than the area of ​​the other parts, the selected area and the target text region are merged to obtain a second merged region; The text included in the second merged region is identified as the transmission object.

7. The method according to any one of claims 3-6, characterized in that, The process of matching the selected range and the page analysis result to obtain the transmission object further includes: If the ratio is less than the first threshold, the transmission object is determined to be an image included in the selection range.

8. The method according to any one of claims 2-7, characterized in that, The step of obtaining the selection range based on the circled trajectory includes: The circumscribed regular polygon of the trajectory data is obtained based on the trajectory data corresponding to the selected trajectory. Use the circumscribed regular polygon as the selection range.

9. The method according to any one of claims 1-8, characterized in that, The second interface also includes a first control; The method further includes: Receive the first trigger operation applied to the first control; In response to the first triggering operation, the type of the transmission object is updated from a first type to a second type, where the first type is different from the second type, and the type of the transmission object includes text type or image type; The third interface is displayed, which includes: the deformed screenshot, the updated transmission object marked on the deformed screenshot, and the service icon of the service corresponding to the updated transmission object.

10. The method according to claim 9, characterized in that, The method further includes: If the object being transmitted is an image that does not contain text, a prompt indicating that the object being transmitted does not contain text is displayed in response to the triggering operation.

11. The method according to any one of claims 1-10, characterized in that, The second interface also includes a second control; The method further includes: Receive a second trigger operation applied to the second control; In response to the second triggering operation, an editing interface for the transmission object is displayed.

12. The method according to any one of claims 1-11, characterized in that, The second interface also includes a third control; The method further includes: Receive a third trigger operation applied to the third control; In response to the third triggering operation, the transmitted object is stored in a preset location.

13. The method according to any one of claims 1-12, characterized in that, The step of performing page analysis on the screenshot to obtain page analysis results includes: The screenshot is processed using a computer vision model to obtain at least one text region and at least one image region.

14. The method according to any one of claims 1-13, characterized in that, The service icon of the service corresponding to the transmission object includes the service icon of the target service. The method further includes: In response to a trigger operation on the service icon of the target service, the interface of the application corresponding to the target service is displayed.

15. An electronic device, characterized in that, The electronic device includes a processor and a memory; the processor is coupled to the memory; the memory is used to store computer program code; the computer program code includes computer instructions, which, when executed by the processor, cause the electronic device to perform the method as described in any one of claims 1-14.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-14.

Images