A data processing method and related apparatus

Abstract

A data processing method applied to a terminal device, the method comprising: receiving a first operation of a user; the first operation being used to select a first display element on a display interface of the terminal device; in response to the first operation, displaying a selection prompt for the first display element and a first AI function control; wherein the first AI function control is used to implement an AI function, the AI function being determined according to at least one of content, type of the first display element and a scene in which the terminal device is located; receiving a selection operation for the first AI function control, and executing the AI function on the first display element. The present application can reduce the operation steps of the user, thereby improving the user experience.

Description

Technical Field

[0001] This application relates to the field of page display technology, and in particular to a data processing method and related apparatus. Background Technology

[0002] With the development of artificial intelligence (AI) technology, various terminal devices are gradually beginning to inherit AI functions, that is, to integrate an AI program at the system layer or application layer to realize the calling of AI functions.

[0003] The typical user process for activating AI is as follows: After launching the AI ​​program, the user selects an object on the interface by drawing or dragging, and then interacts with the AI ​​program through dialogue or text to express their intention to the selected object. The AI ​​program then executes the corresponding AI function. This process requires multiple steps to convey the user's intention to the AI ​​assistant, and it is time-consuming. Summary of the Invention

[0004] This application provides a data processing method and related apparatus that can reduce the number of steps a user takes when using AI, improve interaction efficiency, and thus enhance the user experience.

[0005] In a first aspect, embodiments of this application provide a data processing method applied to a terminal device. The method includes: receiving a first operation from a user; responding to the first operation, selecting a first display element on the display interface of the terminal device and displaying a first AI function control; wherein the first AI function control is used to provide an input entry point for an AI program; receiving a preset operation for the first AI function control, and transmitting the first display element to the AI ​​program so that the AI ​​program performs an AI function on the first display element.

[0006] Through the above embodiments, after the user inputs a first operation to select a first display element on the display interface, the terminal device can respond to the first operation and determine the AI ​​function that the user wants to perform on the first display element based on the relevant information of the first display element or the current usage environment of the terminal device. Then, it displays a first AI function control on the display interface to trigger the AI ​​function. In other words, the first operation can achieve the functions of selecting an object, waking up the AI ​​program, and inputting the AI ​​function to be performed. Therefore, the user only needs to input the first operation and does not need to input the operation of the AI ​​function to be performed. The user can directly obtain the AI ​​function control that may meet their intention from the display interface. If the AI ​​function control meets their needs, the user can quickly execute the AI ​​function by selecting the AI ​​function control, thereby reducing the user's operation steps, improving interaction efficiency, and thus enhancing the user experience.

[0007] In one possible implementation, the method further includes: receiving a text instruction or voice instruction input by a user; the step of passing the first display element to the AI ​​program so that the AI ​​program can perform AI functions on the first display element includes: passing the first display element and the text instruction or voice instruction to the AI ​​program so that the AI ​​program can perform the AI ​​function indicated by the text instruction or voice instruction on the first display element.

[0008] In one possible implementation, the first operation is used to select a first display element on the terminal device's display interface. This first operation can be a selection operation on the first display element, an operation by the user to activate the AI ​​program, or a selection operation on a blank area of ​​the interface. In the latter case, the selection operation on the first display element is equivalent to the user actively selecting it. The operations to activate the AI ​​program and the selection operation on a blank area of ​​the interface do not involve the user directly selecting the first display element; instead, the system automatically selects the object from the display page where the user might perform AI functions (i.e., the first display element). In this case, the first display element can be the page itself or an element selected from the page.

[0009] In one possible implementation, the first display element is the object selected by the first operation; or, the first display element is determined based on the context information of the object selected by the first operation in the display interface; or, the first display element is selected from the display elements in the display interface based on at least one of the type of the display interface, the type of the display elements in the display interface, the content of the display elements in the display interface, and the display size of the display elements.

[0010] In one possible implementation, displaying the second AI function control combination includes: displaying a third AI function control or a folding control; receiving an operation on the third AI function control or the folding control, and unfolding the third AI function control or the folding control to display the second AI function control combination.

[0011] In one possible implementation, the method further includes: displaying a second AI function control in response to the first operation; and executing the AI ​​function indicated by the text or voice command on the first display element upon receiving a selection operation on the second AI function control and a text or voice command input by the user.

[0012] The selection operation for the second AI function control is a direct operation on the second AI function control, and this operation is unrelated to the first display element.

[0013] In one possible implementation, displaying the second AI function control includes: displaying a combination of second AI function controls, the combination of second AI function controls including multiple AI function controls, and different AI function controls in the combination of second AI function controls serving as entry points for AI programs corresponding to different AI agents; receiving a selection operation for the second AI function control includes: receiving a selection operation for the second AI function control among the multiple AI function controls.

[0014] In one possible implementation, the display of multiple AI function controls includes: displaying a third AI function control or a folding control; receiving an operation on the third AI function control or the folding control; and expanding the third AI function control or the folding control to display the multiple AI function controls.

[0015] In one possible implementation, the first AI function control includes a content receiving area; receiving a selection operation for the first AI function control includes receiving a drag operation to drag the first display element to the content receiving area.

[0016] In one possible implementation, the display position of the first AI function control is related to the position of the first display element.

[0017] In one possible implementation, the scenario includes either time or location.

[0018] In one possible implementation, the method further includes:

[0019] A dialog window is displayed, within which the first display element received by the AI ​​program is shown.

[0020] The first AI function control is used to implement AI functions, which are determined based on at least one of the content and type of the first display element and the scene in which the terminal device is located.

[0021] Secondly, this application provides a data processing apparatus for use in a terminal device, the apparatus comprising:

[0022] The receiving module is used to receive the user's first operation;

[0023] The control module is used to respond to the first operation by controlling the display module to select a first display element on the display interface of the terminal device and display a first AI function control; wherein, the first AI function control is used to provide an input entry point for the AI ​​program;

[0024] The receiving module is further configured to receive a selection operation for the first AI function control, and the control module is configured to pass the first display element to the AI ​​program so that the AI ​​program can perform AI functions on the first display element.

[0025] In one possible implementation, the receiving module is further configured to: receive text or voice commands input by the user; the control module is configured to:

[0026] The first display element and the text or voice command are transmitted to the AI ​​program so that the AI ​​program can execute the AI ​​function indicated by the text or voice command on the first display element.

[0027] In one possible implementation, the first operation is a selection operation of a first display element in the display interface of the terminal device; or...

[0028] The first operation is to wake up the AI ​​program; or,

[0029] The first operation is a selection operation on a blank area in the display interface.

[0030] In one possible implementation, the first displayed element is the object selected by the first operation; or...

[0031] The first display element is determined based on the context information of the object selected in the first operation on the display interface; or,

[0032] The first display element is selected from the display elements in the display interface based on at least one of the type of the display interface, the type of the display element in the display interface, the content of the display element in the display interface, and the display size of the display element.

[0033] In one possible implementation, the control module is used to control the display module:

[0034] Display the first AI function control combination, which includes multiple AI function controls. Different AI function controls in the first AI function control combination are the entry points for AI programs corresponding to different AI agents.

[0035] The receiving module is used for:

[0036] A preset operation is received for the first AI function control among the plurality of AI function controls.

[0037] In one possible implementation, the control module is used to control the display module to display a third AI function control or a folding control;

[0038] The receiving module is used to receive operations on the third AI function control or the folding control, and the control module is used to control the display module to unfold and display the third AI function control or the folding control as the first AI function control combination.

[0039] In one possible implementation, the control module is used to control the display module:

[0040] In response to the first operation, a second AI function control is displayed;

[0041] The receiving module is used for:

[0042] Receive selection operations for the second AI function control and text or voice commands input by the user;

[0043] The control module is used to control the first display element to execute the AI ​​function indicated by the text command or voice command.

[0044] In one possible implementation, the control module is used to control the display module:

[0045] Display the second AI function control combination, which includes multiple AI function controls. Different AI function controls in the second AI function control combination are the entry points for AI programs corresponding to different AI agents.

[0046] The receiving module is used for:

[0047] A selection operation is received for the second AI function control among the plurality of AI function controls.

[0048] In one possible implementation, the control module is used to control the display module:

[0049] Display the third AI function control or the collapse control;

[0050] The receiving module is configured to: receive operations on the third AI function control or the folding control;

[0051] The control module is used to unfold the third AI function control or the folded control to display the combination of the second AI function controls.

[0052] In one possible implementation, the first AI function control includes a content receiving area; the receiving module is specifically used for:

[0053] A drag operation is received to drag the first display element to the content receiving area.

[0054] In one possible implementation, the display position of the first AI function control is related to the position of the first display element.

[0055] In one possible implementation, the scenario includes time or location.

[0056] In one possible implementation, the control module is used to control the display module:

[0057] A dialog window is displayed, within which the first display element received by the AI ​​program is shown.

[0058] Thirdly, this application provides a page display device, including: a processor, a memory, and a bus, wherein: the processor and the memory are connected via the bus;

[0059] This memory is used to store computer programs or instructions;

[0060] The processor is configured to call or execute programs or instructions stored in the memory to implement the steps described in the first aspect and any possible implementation of the first aspect.

[0061] Fourthly, this application provides a computer storage medium including computer instructions that, when executed on an electronic device or server, perform the steps described in the first aspect and any possible implementation thereof.

[0062] Fifthly, this application provides a computer program product that, when run on an electronic device or server, performs the steps described in the first aspect and any possible implementation thereof.

[0063] Sixthly, this application provides a chip system including a processor for supporting a device in implementing the functions involved in the foregoing aspects, such as transmitting or processing data involved in the foregoing methods; or, information. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for executing or training the device. The chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0064] To more clearly illustrate the technical methods of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below.

[0065] Figure 1 A schematic diagram of an application architecture provided for an embodiment of this application;

[0066] Figure 2 A schematic diagram of an application architecture provided for an embodiment of this application;

[0067] Figure 3 This is a schematic diagram of a data processing method provided in an embodiment of this application;

[0068] Figures 4 to 12 A schematic diagram of an interface provided for an embodiment of this application;

[0069] Figure 13 A schematic diagram of an apparatus provided in an embodiment of this application;

[0070] Figure 14 This is a schematic diagram of an apparatus provided in an embodiment of this application. Detailed Implementation

[0071] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0072] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0073] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0074] The terms “substantially,” “about,” and similar terms used herein are used as approximations rather than as terms of degree, and are intended to take into account the inherent biases of measurements or calculations known to those skilled in the art. Furthermore, the use of “may” in describing embodiments of the invention refers to “one or more possible embodiments.” The terms “use,” “using,” and “used” used herein are to be considered synonymous with the terms “utilize,” “utilizing,” and “utilized,” respectively. Additionally, the term “exemplary” is intended to refer to an instance or illustration.

[0075] For ease of understanding, the structure of the terminal 100 provided in the embodiments of this application will be illustrated below. See also Figure 1 , Figure 1 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application.

[0076] like Figure 1As shown, terminal 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0077] It is understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the terminal 100. In other embodiments of this application, the terminal 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0078] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0079] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0080] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0081] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0082] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the terminal 100.

[0083] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to enable the function of answering phone calls through a Bluetooth headset.

[0084] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0085] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable music playback through Bluetooth headphones.

[0086] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to enable the shooting function of the terminal 100. The processor 110 and the display screen 194 communicate via the DSI interface to enable the display function of the terminal 100.

[0087] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0088] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge terminal 100, and can also be used for data transfer between terminal 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0089] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the terminal 100. In other embodiments of this application, the terminal 100 may also adopt different interface connection methods or a combination of multiple interface connection methods as described in the above embodiments.

[0090] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the terminal 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0091] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0092] The wireless communication function of terminal 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0093] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0094] The mobile communication module 150 can provide solutions for wireless communication applications including 2G / 3G / 4G / 5G on the terminal 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via the antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via the antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0095] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0096] The wireless communication module 160 can provide solutions for wireless communication applications on the terminal 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0097] In some embodiments, antenna 1 of terminal 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling terminal 100 to communicate with networks and other devices via wireless communication technology. This wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0098] Terminal 100 implements display functions through a GPU, display screen 194, and application processor. The GPU is a microprocessor for page display, connecting the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information. Specifically, one or more GPUs in processor 110 can perform image rendering tasks (such as the rendering tasks related to the image to be displayed in this application), and pass the rendering results to the application processor or other display driver, which triggers the display screen 194 to display video.

[0099] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Minied, MicroLED, Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, terminal 100 may include one or N displays 194, where N is a positive integer greater than 1. The displays 194 can display the target video in the embodiments of this application. In one implementation, terminal 100 can run a camera-related application. When the camera-related application is opened on the terminal, display screen 194 can display a shooting interface, which may include a viewfinder, within which the target video can be displayed.

[0100] Terminal 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0101] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits this electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0102] Camera 193 is used to capture still images or videos.

[0103] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when terminal 100 selects a frequency point, the DSP can perform Fourier transforms on the frequency energy.

[0104] Video codecs are used to compress or decompress digital video. Terminal 100 may support one or more video codecs. Thus, terminal 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0105] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs can enable intelligent cognitive applications in terminals, such as image recognition, facial recognition, speech recognition, and text understanding.

[0106] The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the terminal 100. The external storage card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card.

[0107] Internal memory 121 can be used to store computer executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc. The data storage area may store data created during the use of terminal 100 (such as audio data, phone book, etc.). In addition, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of terminal 100 by running instructions stored in internal memory 121 and / or instructions stored in memory set in the processor.

[0108] Terminal 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0109] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0110] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The terminal 100 can listen to music or make hands-free calls through the speaker 170A.

[0111] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the terminal 100 receives a phone call or voice message, the receiver 170B can be brought close to the listener's ear to hear the voice.

[0112] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Terminal 100 may have at least one microphone 170C. In some embodiments, terminal 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, terminal 100 may have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0113] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0114] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on the display screen 194 or the side edge of the terminal. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Terminal 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, terminal 100 detects the intensity of the touch operation based on pressure sensor 180A. Terminal 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands.

[0115] The electronic device has a screen and a mid-frame, with the screen fixed to the mid-frame.

[0116] In one embodiment, the electronic device has a side button on its mid-frame. A capacitive touch sensor or other touch sensor can be mounted on the surface of the side button to detect user touch gestures. In addition, the electronic device also has at least one pressure sensor 180A corresponding to the side button to sense pressure applied to it.

[0117] In another embodiment, at least a portion of the electronic device's frame can undergo micro-deformation under external force, and at least one pressure sensor is provided for this micro-deformable portion to sense the pressure. For touch recognition, a touch sensor can be placed on the micro-deformable portion to achieve touch detection, or optical ranging can be used. Optical ranging involves emitting light from a light source; when the light is blocked by a finger, the distance to the obstacle is measured, and the gesture and the swipe distance of the gesture are detected by detecting changes in distance.

[0118] In general, electronic devices can detect pressure and touch gestures applied to the bezel, which can be achieved by setting buttons on the bezel or by using the entire bezel as a button.

[0119] The gyroscope sensor 180B can be used to determine the motion attitude of the terminal 100. In some embodiments, the gyroscope sensor 180B can determine the angular velocity of the terminal 100 around three axes (i.e., the x, y, and z axes). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal 100's shake, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the terminal 100 through reverse movement, thus achieving image stabilization. The gyroscope sensor 180B can also be used in navigation and motion-sensing game scenarios.

[0120] The barometric pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal 100 calculates altitude using the air pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0121] The magnetic sensor 180D includes a Hall sensor. The terminal 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover. In some embodiments, when the terminal 100 is a flip phone, the terminal 100 can detect the opening and closing of the flip cover using the magnetic sensor 180D. Then, based on the detected opening and closing state of the cover or the flip cover, features such as automatic flip unlocking can be set.

[0122] The 180E accelerometer can detect the magnitude of acceleration of terminal 100 in various directions (typically three axes). When terminal 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic devices, and is applied to applications such as screen orientation switching and pedometers.

[0123] A distance sensor 180F is used to measure distance. The terminal 100 can measure distance via infrared or laser. In some embodiments, during a shooting scene, the terminal 100 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0124] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The terminal 100 emits infrared light outward through the LED. The terminal 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal 100. When insufficient reflected light is detected, the terminal 100 can determine that there is no object near the terminal 100. The terminal 100 may use the proximity sensor 180G to detect when a user holds the terminal 100 close to their ear for a call, so as to automatically turn off the screen to save power. The proximity sensor 180G can also be used in holster mode and pocket mode for automatic unlocking and screen locking.

[0125] The ambient light sensor 180L is used to sense the ambient light intensity. The terminal 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light intensity. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also work with the proximity sensor 180G to detect whether the terminal 100 is in a pocket to prevent accidental touches.

[0126] The fingerprint sensor 180H is used to collect fingerprints. The terminal 100 can use the characteristics of the collected fingerprints to unlock the device, access application locks, take photos with fingerprints, and answer calls with fingerprints.

[0127] Temperature sensor 180J is used to detect temperature. In some embodiments, terminal 100 uses the temperature detected by temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, terminal 100 reduces the performance of the processor located near temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is below another threshold, terminal 100 heats battery 142 to prevent abnormal shutdown of terminal 100 due to low temperature. In still other embodiments, when the temperature is below yet another threshold, terminal 100 boosts the output voltage of battery 142 to prevent abnormal shutdown due to low temperature.

[0128] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of terminal 100, in a different position than display screen 194.

[0129] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure signals. In some embodiments, the bone conduction sensor 180M can also be incorporated into headphones to form bone conduction headphones. The audio module 170 can parse the voice signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 180M to realize voice functionality. The application processor can parse heart rate information from the blood pressure signals acquired by the bone conduction sensor 180M to realize heart rate detection functionality.

[0130] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Terminal 100 can receive button input and generate key signal inputs related to user settings and function control of terminal 100.

[0131] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0132] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0133] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the terminal 100. The terminal 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. These multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The terminal 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the terminal 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal 100 and cannot be separated from the terminal 100.

[0134] The software system of terminal 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses the layered architecture Android system as an example to illustrate the software structure of terminal 100.

[0135] Figure 2 This is a software structure block diagram of terminal 100 according to an embodiment of this disclosure.

[0136] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0137] The application layer can include a series of application packages.

[0138] like Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0139] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0140] like Figure 2 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0141] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0142] Content providers store and retrieve data, making that data accessible to applications. This data can include videos, images, audio, phone calls made and received, browsing history and bookmarks, phone books, etc.

[0143] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0144] The phone manager is used to provide communication functions for terminal 100. For example, it manages call status (including connection, hang-up, etc.).

[0145] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0146] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0147] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0148] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0149] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0150] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0151] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0152] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0153] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0154] A 2D graphics engine is a graphics engine for 2D drawing.

[0155] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0156] With the development of artificial intelligence (AI) technology, various terminal devices are gradually beginning to inherit AI functions, that is, to integrate an AI program at the system layer or application layer to realize the calling of AI functions.

[0157] The typical user process for activating AI is as follows: After launching the AI ​​program, the user selects an object on the interface by drawing or dragging, and then interacts with the AI ​​program through dialogue or text to express their intention to the selected object. The AI ​​program then executes the corresponding AI function. This process requires multiple steps to convey the user's intention to the AI ​​assistant, and it is time-consuming.

[0158] To address the aforementioned problems, embodiments of this application provide a data processing method. For ease of understanding, the data processing method provided by embodiments of this application will be specifically described in conjunction with the accompanying drawings and application scenarios.

[0159] Reference Figure 3 , Figure 3 This is a flowchart illustrating a data processing method according to an embodiment of this application, which includes:

[0160] 301. Received the user's first operation.

[0161] In this process, the user can perform a first operation on the terminal device, and the terminal device can receive the first operation.

[0162] In existing technologies, in order to input the display elements in the display interface into the AI ​​program to perform the expected function, it is necessary to select the object, wake up the AI ​​program, input the selected object into the AI ​​program, and input the AI ​​function to be performed. Due to the excessive number of steps, the operation is cumbersome and the user experience is poor. This application embodiment can reuse the above-mentioned multiple steps, thereby reducing the number of operation steps. Specifically, after the user inputs a first operation to select a first display element on the display interface, the terminal device can respond to the first operation and determine the AI ​​function that the user wants to perform on the first display element based on the relevant information of the first display element or the current usage environment of the terminal device (for example, when the user sees a product image, they expect to search for / buy the product in a shopping app; when viewing an article / article link or receiving a document through instant messaging software, they expect to summarize the information). A second AI function control for triggering the AI ​​function is then displayed on the display interface. In essence, the first operation can achieve the functions of selecting an object, activating the AI ​​program, and inputting the AI ​​function to be performed. Therefore, the user only needs to input the first operation and does not need to input the operation of the AI ​​function to be performed. They can directly obtain an AI function control that may meet their intention from the display interface. If the AI ​​function control meets their needs, they can quickly execute the AI ​​function by selecting the AI ​​function control, thereby improving the user experience.

[0163] The first operation can be used to select a first display element on the terminal device's display interface. This first operation can be a selection operation targeting the first display element, or it can be an operation not directly related to the first display element, but the electronic device can infer the user's intention to select the first display element based on the first operation. The first operation can be a selection operation targeting the first display element, an operation used by the user to activate the AI ​​program, or a selection operation targeting a blank area on the interface. Selecting the first display element is equivalent to the user actively selecting it. Operations to activate the AI ​​program and selection operations targeting a blank area on the interface do not involve the user directly selecting the first display element; instead, the system automatically selects the object from the display page where the user might perform AI functions (i.e., the first display element). In this case, the first display element can be the page itself or an element selected from the page.

[0164] In this embodiment, the display page can be the page that the application (or system desktop) is visually displaying (which can be called the screen interface), or it can be a page that has not yet been displayed but can be displayed by the user through swiping or switching operations (which can be called the application interface). For example, the current screen interface only includes the content displayed on the current screen, while the application interface includes all content of the current application interface, including both what is displayed on the screen and what is not displayed on the screen. Taking news viewing as an example, the current screen interface is the part of the news paragraph that is being displayed, while the current application interface includes the entire news article.

[0165] The first operation in the embodiments of this application will now be described in detail.

[0166] In one possible implementation, the first operation is a selection operation for a first display element in the display interface of the terminal device.

[0167] Selection operations can be touch operations, where the user has a touch position when performing a preset screen gesture. When an electronic device renders displayed content, each element has its own display position. Therefore, the first display element can be determined based on the overlap between the touch position and the display position of the touch operation. If the touch position overlaps with the display position of a certain display element, the touch gesture is considered to have acted on that display element, and that display element is the first display element. It can be understood that the touch position here is a general term; when the preset gesture is a long press, the long press position is the touch position; when the preset gesture is a selection position, the touch position is the area within the selection trajectory; when the preset gesture is a two-finger pinch, the intersection of the extended lines of the two-finger trajectory can be taken as the touch position.

[0168] When a display location has multiple display elements located on different layers, the display element on the topmost layer is usually designated as the first display element because users cannot see the obscured bottom layers and therefore will not generate any interactive intent.

[0169] In a specific implementation, a center point can be determined based on the touch gesture. The element to be displayed is then designated as the first element based on whether this center point falls within its display area. For example, a long press gesture covering a cluster of pixels allows the identification of the cluster's geometric center point. Similarly, a selection gesture first fits a closed shape based on the selection trajectory, then determines the geometric center point of that shape. Once the geometric center point is determined, it can be used to identify the first element based on whether it falls within the display area of ​​another element.

[0170] Alternatively, a touch area can be defined based on the touch gesture, and the first display element can be determined based on the overlap between the touch area and the display element. The touch area can be the region containing all the pixels touched by the user, or it can be a closed graphic region formed by the touched pixels. The former is suitable for long-press gestures, and the latter is suitable for selection gestures. If the overlap between the touch area and a display element exceeds a preset value, that display element can be designated as the target gesture. For selection gestures, multiple display elements may be selected. If the overlap between multiple display elements and the touch area is all higher than a preset value, the display element with the largest area can be designated as the first display element.

[0171] In one possible implementation, the first operation is the operation of waking up the AI ​​program.

[0172] Users can activate the AI ​​program through various actions. For the terminal device itself, there are multiple ways to launch the AI ​​program, such as through preset screen gestures, physical buttons or pressure-sensitive buttons, back gestures, or clicking the AI ​​program's application icon. In this embodiment, the AI ​​program is activated through a preset screen gesture.

[0173] In one possible implementation, the operation to wake up the AI ​​program is a long-press screen gesture. In another design, the operation to wake up the AI ​​program is a selection gesture on the screen. Of course, in practical applications, other screen gestures can also be used. Alternatively, it can be a combination of multiple screen gestures, such as long-press to select, long-press to throw, double-tap to select, and many other combinations of two, three, or more screen gestures.

[0174] In one possible implementation, the first operation is a selection operation on a blank area in the display interface.

[0175] When the first operation is a long press or other narrow-range operation, there may be a situation where the touch location is a blank area. In this case, it is difficult to determine the first display element using the methods described above. Similarly, it is difficult to determine the first display element when waking up an AI program. In such cases, the current screen interface or the current application interface (or a portion of the page) is used as the first display element.

[0176] In one possible implementation, the first display element can be text content, image content (e.g., a complete image or a portion of an image), a card, audio data, etc. The first display element is the object the user is expected to interact with, determined from the display elements of the interface. It can be an object displayed on the screen, such as an image, text, a control or a component composed of multiple controls, a card / widget, or even an element within an object. For example, if the current page is a news article combining text and images, the first display element could be an image inserted within the news article or text from the news article itself. As another example, if an image contains multiple display elements (e.g., people, animals, flowers), then those elements can also serve as the first display element.

[0177] Among the types of first operations listed above, if the first operation is a selection operation for a first display element in the display interface of the terminal device, then the first display element can be the selection object of the first operation, or determined based on the selection object of the first operation and the context information of the selection object in the display interface. If the first operation is an operation to wake up an AI program or a selection operation for a blank area in the display interface, then the first display element can be selected from the display elements in the display interface based on at least one of the type of the display interface, the type of display elements in the display interface, and the content of the display page.

[0178] Reference Figure 4 , Figure 4 The first operation is an illustration of the selection operation of the first display element in the display interface of the terminal device. Figure 4 This demonstrates the selection of the first display element 401 under various interaction modes. Figure 4 In element 'a', if a long press on a linked article overlaps with the forwarded article based on the position of the long press, then the forwarded article (article link) is displayed as the first element 401. Figure 4 In option b, if the user selects an image within a received message bubble, the image within the selected shape will be displayed as the first element 401. Figure 4 In C, when a user clicks on an image to view it, and selects a portion of the image on the image viewing page, the image in the upper right corner becomes the first displayed element 401.

[0179] Since the first action is a screen gesture, and screen gestures may have existing interaction logic within the application's memory. For example, Figure 4 In dialog box A, long-pressing on the dialogue content brings up several controllable actions that can be performed on that object. For example, long-pressing on text allows you to select text.

[0180] Based on this, the first display element 401 can also be determined according to the interaction result of the first operation within the application. A typical scenario is that when viewing text content such as news, if the first operation is a long press, the system or the application itself will trigger text selection for the long-pressed text to facilitate functions such as copying after selecting the text. After triggering text selection, a selection box is usually added to several texts near the touch position. In this case, instead of just recognizing a single text near the touch position as the first display element 401, several texts within the selection box are recognized as the first display element 401.

[0181] like Figure 5a As shown, after the user long-presses part of the text, the electronic device responds to the long-press action and selects four nearby texts. The four selected texts then serve as the first display element 401.

[0182] When the first operation cannot determine the first display element (e.g., the wake-up operation of an AI program or the selection operation for a blank area), the current interface is first screen-recognized, and then the first display element is determined based on the screen recognition result, thereby determining the AI ​​function.

[0183] Specific screen recognition may include, but is not limited to, recognition of at least one of the following: the type of the displayed page, the type of displayed elements within the page, the size of the displayed elements (e.g., the element occupying the largest proportion of the current interface), and preset information. AI functions are then determined based on these screen recognition results. For example, if the page type is a chat interface and there are many unread messages, the recommendation function could be a chat summary. Preset information recognition could involve identifying the information in Table 1.

[0184] Figure 5b In the middle, users can activate the AI ​​program by long-pressing the bottom navigation bar or the side button, and the AI ​​program will then perform screen recognition. Figure 5b If date information [next Saturday at 10:00 AM] and location information [XX location] are identified in (a), then these two pieces of information are determined as the first display element 401, and an AI function is determined for each of them. The corresponding AI function control 601 is then displayed. The two function controls 601 correspond to different first display elements. Figure 5b In (b), which is the image viewing interface, based on the recognition of objects in the image, different AI function controls 601 are recommended for different display elements 401, and the AI ​​function controls 601 are displayed near the objects. In the image, in order to highlight the display of the first display element 401, the first display element is outlined.

[0185] In some designs, instead of directly displaying the AI ​​function controls, the first identified display element is simply marked, and the user interacts with the marked first display element to select the actual first display element, and then the AI ​​function control corresponding to the actual first display element is displayed.

[0186] In summary, when determining the first display element 401, it can be directly obtained from the displayed content of the current page, such as an image on the page. Alternatively, it can be determined based on the page's displayed content combined with screen recognition, such as identifying which element within an image corresponds to which part mentioned above. It can also be determined based on the interaction response to the first operation, such as the text selected after a long press or selection.

[0187] 302. In response to the first operation, select a first display element on the display interface of the terminal device and display a first AI function control; wherein the first AI function control is used to provide an input entry point for the AI ​​program.

[0188] Optionally, in response to the first operation, a selection prompt for the first display element and a second AI function control may be displayed; wherein the second AI function control is used to implement an AI function, which is determined based on at least one of the content and type of the first display element and the scene in which the terminal device is located.

[0189] In one possible implementation, after the first display element is selected, a selection tooltip for the first display element can be displayed. For example, the selection tooltip could be a highlight effect for the first display element, changing the color of the first display element to a different color from the surrounding colors, enlarging the display of the first display element, a marker indicating that the first display element is selected, or other ways to indicate that the first display element is selected by changing the display state of the first display element.

[0190] In one possible implementation, after the first display element is selected, an AI function can be determined based on at least one of the content types of the first display element, which is the AI ​​function that the user expects to perform on the first display element.

[0191] In other words, after selecting the first display element, a trigger control for the AI ​​function (e.g., the second AI function control in this embodiment) can be automatically presented to the user. This AI function is one that the user is likely to perform on the first display element. The AI ​​function that the user expects to perform on the display element is usually related to the information of the display element. The information of the display element can be its specific content or type. For example, if the display element is text, the user may prefer to perform AI functions such as summary extraction and text optimization on the text; if the display element is an image, the user may prefer to perform AI functions such as image noise reduction and beautification on the image.

[0192] Therefore, based on the information of the displayed element selected by the user (e.g., the specific content or type of the displayed element), it is possible to determine which AI function(s) to recommend to the user.

[0193] The following provides a correspondence between the types of display elements and the AI ​​functions they correspond to. For example, if the first display element is an article link, the user typically needs a summary or abstract; for an image-based element, a description of the image content might be required. Table 1 provides some examples of the correspondence between the types of the first display elements and the expected AI functions.

[0194] Table 1

[0195] The first displayed element image type AI Functions Schedule-related (time, date, etc.) Create schedule cards Address type (address or address description) Create calendar cards, location navigation Image Smart products, shopping link redirection Text-based (full text, article link) Abstract extraction, translation, and AI editing Document types (PDF, Word, etc.) Abstract Extraction Chat conversation Reply Recommendation Settings Automatic execution Sensitive information (birthday, etc.) Recording memories

[0196] As shown in Table 1, when the first displayed element is a schedule element (e.g., date and time), the expected AI function is creating a schedule card. When the first displayed element is an address element (e.g., address or address description), the expected AI function is creating a schedule card or providing location navigation. When the first displayed element is an image element, the expected AI function is connecting to a smart product or a shopping link. When the first displayed element is text-based (e.g., full text, article link), the expected AI function is summarizing, translating, or AI editing. When the first displayed element is a file element (e.g., PDF, Word, etc.), the expected AI function is summarizing. When the first displayed element is a chat dialogue, the expected AI function is replying with recommendations. When the first displayed element is a settings item, the expected AI function is automatic execution. When the first displayed element is sensitive information (e.g., birthday), the expected AI function is recording and remembering.

[0197] When determining the AI ​​function that the user intends to perform, the description information of the first display element can be passed to the AI ​​program. For example, at least one of the content or type of the first display element can be passed to the AI ​​program, which then determines the AI ​​function to be performed on the first display element based on at least one of its content and type. Subsequently, a second AI function control for implementing that AI function can be displayed.

[0198] In some embodiments, after selecting the first display element, the specific content of the first display element, its description, or the scene of the terminal device can be passed to the AI ​​program. For example, the specific content of the first display element (e.g., an image, or a portion of an image element obtained through screen recognition) can be passed to the AI ​​program. When the first display element is a link address, its specific content can be the complete content obtained through the link address, such as a complete article obtained through an article link.

[0199] In some designs, after determining the first display element, instead of immediately passing its specific content to the AI ​​program, the descriptive information of the first display element can be passed. This descriptive information can include the type of the first display element, such as plain text, image, or link. Passing the descriptive information only requires a quick command, and the AI ​​program can respond immediately. In contrast, directly passing the specific content of the first display element often involves transmitting its HTML address, storage address, or link URL, requiring the AI ​​program to perform further actions before providing a response. Therefore, passing the descriptive information first achieves a faster response.

[0200] In some embodiments, when determining the AI ​​function that the user expects, the electronic device can also determine it based on scene information of the physical environment in which the terminal device is located. Scene information can be acquired by the terminal device through sensors or a network. Scene information includes location information, time information, weather information, memo information, to-do items, etc. Optionally, the location information mentioned here can be location information after secondary transformation of map information, such as being near a subway station or inside a shopping mall.

[0201] For example, when the AI ​​function cannot be determined based on the current interface or page, it can be recommended based on context information. For instance, when waking up an AI program on the desktop or lock screen, it's difficult to determine the AI ​​function based solely on the content displayed on the desktop / lock screen. In this case, the AI ​​function can be determined based on context information.

[0202] In implementing AI-powered recommendations based on contextual information, the bottom navigation bar is transparent when there are no recommendations; when recommendable content is initially identified, the bottom navigation bar can turn colored and be accompanied by animated prompts; furthermore, the AI ​​function controls for the recommendation function can be directly displayed.

[0203] Figure 6 An interface for making recommendations based on scene information is shown. Figure 6 In (a), there are currently no corresponding features recommended. Figure 6 In (b), based on scene information detection, the AI ​​program identifies recommendable functions. At this point, if the AI ​​program is activated by long-pressing the bottom navigation bar or in other ways, the following will appear: Figure 6 The interface of (c). Figure 6 In (c), after waking up the AI ​​program, the AI ​​function control 601 is displayed. Figure 6 In (d), clicking the AI ​​function control 601 will directly execute the function of the AI ​​function control 601 and display the execution result as a card. Figure 6 (d) is an illustration of showing the payment code.

[0204] In this embodiment of the application, after the AI ​​function is determined, the AI ​​function control corresponding to the AI ​​function (e.g., the second AI function control) can be displayed, or a first AI function control combination including multiple AI function controls can be displayed, and the user can select from the multiple AI function controls included in the first AI function control combination.

[0205] In one possible implementation, there are multiple AI functions that the user expects. For example, if the first display element is an article, the expected AI functions include: text translation, visual search, article reading, and summary. Therefore, multiple AI function controls, including a second AI function control, can be displayed.

[0206] In one possible implementation, one of the multiple AI function controls (e.g., a third AI function control, which can be a second AI function control or another AI function control besides the second AI function control) can be displayed first, or a collapse control can be displayed. Then, an operation on the third AI function control or the collapse control can be received, expanding the third AI function control or the collapse control to display the multiple AI function controls. Furthermore, the user can select one of the multiple AI function controls (e.g., the second AI function control).

[0207] Figure 7The display interface after activating the AI ​​program is shown. After determining that the first display element 401 is an article link, it is determined that the article link corresponds to multiple AI function controls (e.g., summary, text translation, view search, etc.). One of the AI ​​function controls (e.g., summary) can be displayed. The user can display more AI function controls by interacting with the summary, for example... Figure 7 In the middle, swiping up on the display area of ​​the control corresponding to the summary will display more AI function controls 702.

[0208] In one possible implementation, the display position of the AI ​​function control is related to the position of the first display element.

[0209] The AI ​​function control can be displayed near the position of the first display element. For example, after the first display element is selected, function controls related to the first display element 401 (such as forwarding, favorites, etc.) are often displayed. The AI ​​function control can also be displayed alongside other controls as a function control related to the first display element 401, that is, the AI ​​function control is integrated into the display area of ​​the function control related to the first display element 401.

[0210] For example, Figure 8 In the context of the interface, the first action is a long press. The terminal device responds by displaying a UI component containing multiple controls such as forward, favorite, and delete. AI function controls can be integrated into this UI component, allowing them to be arranged alongside the forward, favorite, and delete controls. To avoid disrupting the layout of existing controls within the UI component, the function controls can be positioned according to the existing layout. For example, if at least one control within the UI component is arranged in a row, a grid is added at the end for the AI ​​function controls; similarly, if at least one control within the UI component is a table column, a new row is added at the end to display the AI ​​function controls. Figure 8 In the UI component, if the existing controls are arranged in a row, a new column or grid is added at the end of the arrangement to display the AI ​​function controls.

[0211] In one possible implementation, to highlight the AI ​​function controls, they can be highlighted with color or use other visual schemes that distinguish them from existing controls within the responsive UI components. These include, but are not limited to, bolding, italics, underlining, and colored fonts.

[0212] 303. Upon receiving a selection operation for the second AI function control, execute the AI ​​function on the first display element.

[0213] After the second AI function control is displayed, the user can interact with it to execute its AI functions. When the user interacts with the second AI function control, if the first display element (e.g., its description) has not been passed to the AI ​​program before, the terminal device can pass the first display element to the AI ​​program. If the content of the first display element has already been passed to the AI ​​program, the AI ​​program can be triggered to execute its AI functions.

[0214] In this way, after activating the AI ​​program, users no longer need to input commands into the AI ​​program. Instead, they can directly obtain at least one functional control on the screen that may meet their intentions. If the functional control meets their needs, they can quickly realize their actual intentions by clicking on the functional control without interacting with the AI ​​program again. This allows the first display element to be passed to the AI ​​program, enabling the AI ​​program to achieve rapid interaction without leaving the current page after activation, and this interaction can be as close as possible to the user's intentions.

[0215] In one possible implementation, a selection operation for the second AI function control can be received, and an AI function can be performed on the first display element, for example, a click operation.

[0216] The second AI function control is used to implement a specific AI function. However, in some cases, the AI ​​function control recommended to the user may not meet the user's expectations (that is, it is not the type of AI function that the user wants to perform).

[0217] In one possible implementation, in addition to displaying the second AI function control, an AI control serving as the input entry point for the AI ​​program (e.g., the first AI function control in this embodiment) can also be displayed. The difference between the first and second AI function controls is that the first AI function control does not directly recommend specific AI functions to the user, but instead provides an input entry point for the AI ​​program. After the user selects the first AI function control, the electronic device automatically inputs the selected first display element into the AI ​​program corresponding to the first AI function control. Since the first AI function control functions as the input of the first display element into the AI ​​program, it can also be called a transfer control.

[0218] Furthermore, when users interact with the first AI function control to input the first display element into the AI ​​program, they can also provide signal commands, which include the functions that the user expects to perform on the first display object. For example, the user inputs a signal command by long-pressing the first AI function control, and when the long-press gesture is released, the first display element is passed to the AI ​​program.

[0219] In one implementation, after the user selects the first AI function control, the electronic device can also display an AI dialog box, in which the user can input voice or text commands. In addition, the AI ​​dialog box can also display the AI ​​program's response to the user's input voice or text commands.

[0220] In this embodiment, after selecting the first display element, the first AI function control can be automatically displayed. The first AI function control serves as the input entry point for the AI ​​program, without requiring additional user operation to open the input entry point for the AI ​​function. The first display element can be quickly input into the AI ​​program through the first AI function control, without having to drag the first display element into the AI ​​program, thus avoiding the interruption that may be caused by dragging over long distances and the difficulty of long-distance operation.

[0221] In one possible implementation, in response to the first operation, a first AI function control can be displayed; this first AI function control serves as an input entry point for the AI ​​program to receive object input through the input interface. After the interface displays the first AI function control, the electronic device detects a preset operation (e.g., a selection operation) on the first AI function control and passes the first display element to the AI ​​program as input for the AI ​​program.

[0222] Multiple first AI function controls can be displayed, and these controls can be provided by different AI programs or by the same AI program. For example, an AI program may include multiple AI agents, with each first AI function control corresponding to a different AI agent, serving as the input entry point for each AI agent. This allows the first display element to be directly passed to each AI agent through the first AI function controls.

[0223] Regarding AI agents: An AI agent is a subroutine of an AI program, based on its general capabilities and trained using domain-specific knowledge. It can be considered a subroutine of the AI ​​program. During user interaction, the AI ​​agent can be further trained based on user data to improve its capabilities. AI agents can be predefined by the AI ​​program or by the user based on the program's capabilities. For example, an AI program can predefine a meeting assistant AI agent specifically for use in meeting scenarios, enabling functions such as meeting minutes and summaries. Alternatively, a user can create a dialogue AI agent specifically for English speaking practice and name it "Speaking Practice." These two agents utilize different capabilities of the AI ​​program. The AI ​​program itself (i.e., its basic capabilities) can also be considered an AI agent, referred to as the main agent, with the voice dialogue modality entry point corresponding to the main agent. Third-party applications can provide AI capabilities and train agents, exposing the agent's interface to the AI ​​program. For example, a homework tutoring application might provide a homework grading agent.

[0224] In one implementation, the electronic device detects a preset operation on the first AI function control, such as clicking the first AI function control, and the electronic device passes the first display element to the AI ​​program. At this time, it is usually passed to the main agent of the AI ​​program for processing by default.

[0225] In another implementation, the AI ​​program includes not only the main AI agent but also several other AI agents. Therefore, the first AI function control, besides serving as the input entry point for the main AI agent, can also act as a folding control. When another preset operation is detected, the input entry points for multiple AI agents are displayed. The user then interacts with these input entry points to pass the first display element to each AI agent. For example, clicking the first AI function control passes the first display element to the main AI agent; long-pressing the first AI function control displays the input entry points for multiple AI agents on the screen. The electronic device detects the click operation on the input entry point and passes the first display element to the corresponding AI agent as its input. It is understood that the aforementioned click, long-press, and other operations are merely examples and can be implemented using other touch gestures, air gestures, or other interactive operations; there are no limitations on this.

[0226] In existing technologies, when inputting display elements into the entry point of an AI program, a long drag is often required, resulting in a poor user experience. In this embodiment, since the first operation has already selected the first display element, the first AI function control can be selected directly without long drags, thereby improving the user experience.

[0227] In this embodiment, the first AI function control can also be called a transmission control. Through the transmission control, a first display element can be passed to the AI ​​program, triggering a dialogue within the AI ​​program and displaying a dialogue window. The user can then express their intentions by inputting text or voice commands within the dialogue window. After displaying the transmission control, the user can interact with it through clicking, double-clicking, or other interactive methods. The terminal device can then pass the first display element to the AI ​​program. For example, if the first display element is an image stored / cached locally, the image's storage address is passed to the AI ​​program; if it is plain text, the plain text is directly passed to the AI ​​program. Upon receiving the first display element, the AI ​​program displays the AI ​​dialogue window and can display the received first display element within the AI ​​dialogue window. The AI ​​dialogue window supports user input, such as providing a text / voice input switching control for user input. The user communicates their intentions by interacting with the AI ​​program through input.

[0228] Figure 9 In the middle, to the right of the second AI function control, a transfer control 901 (that is, the first AI function control) is displayed. Interacting with this transfer control 901 allows the first display element 401 to be passed to the AI ​​program or AI agent. When the transfer control 901 is clicked, it indicates that the currently recommended AI function does not meet the user's actual needs. Therefore, when passing the first display element 401 to the AI ​​program and displaying the dialog window, the recommended function control and transfer control 901 are no longer displayed on the interface; instead, an input control is displayed. Figure 9 In the center, the three controls at the bottom are, in order, keyboard input, voice input, and visual input. When interacting with the AI ​​program via keyboard and voice input, the user can input their intent based on the transmitted first display element 401 to perform a certain operation on the first display element 401. Visual input is used to allow the AI ​​program to break out of the current environment based on the first display element 401 and reselect the first display element 401 through visual input, providing the AI ​​program with general capabilities.

[0229] Figure 9 Only the main AI agent's transfer control 901 is shown. In some designs, multiple transfer controls 901 can be set, each corresponding to a different AI agent. For example, in addition to the main AI agent, the AI ​​program includes two other AI agents; in this case, three transfer controls 901 can be displayed, corresponding to the main AI agent and the two other AI agents respectively. Typically, the main AI agent is placed at the front by default, and the other AI agents can be ordered according to preset rules, such as last usage time or usage frequency.

[0230] Figure 10In the middle, two other transmission controls 901 are also displayed to the right of the second AI function control, which are respectively pointed to different AI agents.

[0231] In another design, the electronic device can include multiple AI programs, and the transmission control can be used to point to the AI ​​agent of the AI ​​program (usually the AI ​​program integrated into the system), or to different AI programs (for example, one transmission control for one third-party AI program). In this way, the first display element can be passed to both the AI ​​agent and different AI programs.

[0232] Figure 10 In this design, AI agents from different AI applications are showcased in a centralized manner. In another design, referencing... Figure 11 Alternatively, AI agents can be grouped according to their respective applications. This involves displaying icons 1001 for different AI programs on the right side of the function (i.e., through a collapsing control). After interacting with the AI ​​program icon, the icon of the AI ​​agent is displayed as a transfer control 901 for the AI ​​agent. Clicking the transfer control 901 will transfer the first displayed element to the corresponding AI agent.

[0233] Figure 11 One interface change is illustrated, where the user can select the first display element 401 to enter the AI ​​program wake-up interface. This interface displays functional controls related to the first display element 401 (e.g., forwarding, favorites, etc.). Additionally, the interface displays a second AI functional control 601 and multiple AI program collapsible controls 1001. Long-pressing the collapsible control 1001 expands it into multiple AI controls 901 (i.e., entry controls for several AI agents within the AI ​​program). These AI controls 901 can be arranged in a preset order, such as by user frequency. After selecting one AI control (the first AI functional control) from the multiple AI controls 901, the user can pass the first display element to the corresponding AI agent for processing.

[0234] Figure 10 and Figure 11 In the design, the AI ​​function control is styled as an icon, but other designs can also use text, or a combination of icons and text.

[0235] Figure 11 The interface changes can serve as an example. In another interface change, when a user long-presses an AI program icon, the AI ​​function controls and other AI program icons can disappear, displaying only the AI ​​program icon and its list of AI agents. The agents can be arranged around the AI ​​program icon. And there are other changes to the display of AI agents on the interface.

[0236] In one possible implementation, the second AI function control includes a content receiving area; it can then receive drag operations where the first display element is dragged to this content receiving area. Considering that users may be more accustomed to drag-and-drop operations, if the recommended AI function does not meet user needs, the user can perform a drag operation on the first display element. Upon detecting the drag operation, a larger AI receiving area can be displayed. Thus, the user only needs to drag the first display element to this AI receiving area and release it to complete the transfer of the first display element. This method reduces the drag distance by expanding the receiving area, changing the original small AI icon receiving the first display element to displaying a large AI receiving area. See [reference needed] for details. Figure 12 As shown, Figure 12 The receiving area 1201 is shown.

[0237] Furthermore, embodiments of this application also provide a data processing method flow, including:

[0238] Step 1: Receive the user's first operation; the first operation is used to select a first display element on the display interface of the terminal device;

[0239] For a detailed description of step 1, please refer to the description of step 301 in the above embodiment. The similarities will not be repeated here.

[0240] Step 2: In response to the first operation, display the first AI function control; the first AI function control is used to provide the entry point for the AI ​​program;

[0241] Step 3: Upon receiving the selection operation for the first AI function control and the text or voice command input by the user, execute the AI ​​function indicated by the text or voice command on the first display element.

[0242] In one possible implementation, the second AI function control is used to implement a specific AI function. If the AI ​​function space recommended to the user does not meet the user's expectations (that is, it is not the type of AI function that the user wants to execute), in this case, an AI control (e.g., the first AI function control in the embodiments of this application) can also be displayed as the entry point of the AI ​​program. The AI ​​program entry point can receive text commands or voice commands from the user, and the text commands or voice commands can express the AI ​​function that the user expects.

[0243] In this embodiment, after selecting the first display element, the entry point of the AI ​​program can be automatically displayed without requiring additional user operation to open the input entry point of the AI ​​function. After the user inputs the command, they can quickly execute the corresponding AI function on the selected display element by selecting the function control, without having to perform the operation of passing the first display element to the AI ​​program. This enables the AI ​​program to achieve rapid interaction without leaving the current page after it is enabled.

[0244] In one possible implementation, in response to the first operation, a first AI function control can be displayed; the first AI function control is used to provide an entry point for the AI ​​program; thereafter, a selection operation for the first AI function control and a text command or voice command input by the user can be received, and the AI ​​function indicated by the text command or voice command can be executed on the first display element.

[0245] AI function controls can include entry points for AI programs corresponding to multiple AI agents. Therefore, multiple AI function controls can be displayed, with each AI function control serving as the entry point for a different AI program corresponding to a different AI agent. Subsequently, a selection operation can be received for the first AI function control among these multiple AI function controls.

[0246] This application also provides a data processing apparatus, which may be a terminal device, see reference. Figure 13 , Figure 13 This application provides a schematic diagram of the structure of a data processing apparatus, as shown in the embodiments. Figure 13 As shown in the figure, the data processing device 1300 includes:

[0247] Receiver module 1301 is used to receive the user's first operation;

[0248] The receiving module 1301 is also used to receive a selection operation for the first AI function control;

[0249] For a detailed description of the receiving module 1301, please refer to the descriptions in steps 301 and 303, which will not be repeated here.

[0250] The control module 1302 is configured to respond to the first operation by controlling the display module to select a first display element on the display interface of the terminal device and display a first AI function control; wherein the first AI function control is used to provide an input entry point for the AI ​​program; and to pass the first display element to the AI ​​program so that the AI ​​program can perform AI functions on the first display element.

[0251] For a detailed description of the control module 1302, please refer to the descriptions related to data processing in steps 302 and 303, which will not be repeated here.

[0252] For a detailed description of the display module 1303, please refer to the description related to display in step 302, which will not be repeated here.

[0253] The receiving module 1301 can be... Figure 1 The sensor module 180 in the described terminal 100, and the receiving module 1301, can detect user operations (e.g., a first operation, a selection operation for the second AI function control, etc.) and transmit the operation information to the control module 1302. The control module 1302 can be... Figure 1 The processor 110 in the described terminal 100, wherein the display module 1303 may be... Figure 1 The display screen 194 is described in the embodiment. The processor 110 can analyze and process user operation information (e.g., information about a first operation) and control the display module 1303 to display corresponding content based on the processing result (e.g., displaying a selection prompt for the first display element and a second AI function control in response to the first operation). In addition, the control module 1302 can also execute AI functions.

[0254] This application's embodiments reuse multiple steps to reduce operational steps. Specifically, after the user inputs a first operation to select a first display element on the display interface, the terminal device can respond to the first operation and determine the AI ​​function the user wishes to perform on the first display element based on relevant information of the first display element or the current usage environment of the terminal device. A second AI function control for triggering the AI ​​function is then displayed on the display interface. Essentially, the first operation can achieve the functions of selecting an object, activating the AI ​​program, and inputting the desired AI function. Therefore, the user only needs to input the first operation, without needing to input the specific AI function to be performed, and can directly obtain an AI function control that may meet their intent from the display interface. If the AI ​​function control meets their needs, the user can quickly execute the AI ​​function by selecting it, thereby improving the user experience.

[0255] In one possible implementation, the receiving module is further configured to: receive text or voice commands input by the user; the control module is configured to:

[0256] The first display element and the text or voice command are transmitted to the AI ​​program so that the AI ​​program can execute the AI ​​function indicated by the text or voice command on the first display element.

[0257] In one possible implementation, the first operation is a selection operation of a first display element in the display interface of the terminal device; or...

[0258] The first operation is to wake up the AI ​​program; or,

[0259] The first operation is a selection operation on a blank area in the display interface.

[0260] In one possible implementation, the first displayed element is the object selected by the first operation; or...

[0261] The first display element is determined based on the context information of the object selected in the first operation on the display interface; or,

[0262] The first display element is selected from the display elements in the display interface based on at least one of the type of the display interface, the type of the display element in the display interface, the content of the display element in the display interface, and the display size of the display element.

[0263] In one possible implementation, the control module is used to control the display module:

[0264] Display the first AI function control combination, which includes multiple AI function controls. Different AI function controls in the first AI function control combination are the entry points for AI programs corresponding to different AI agents.

[0265] The receiving module is used for:

[0266] A preset operation is received for the first AI function control among the plurality of AI function controls.

[0267] In one possible implementation, the control module is used to control the display module to display a third AI function control or a folding control;

[0268] The receiving module is used to receive operations on the third AI function control or the folding control, and the control module is used to control the display module to unfold and display the third AI function control or the folding control as the first AI function control combination.

[0269] In one possible implementation, the control module is used to control the display module:

[0270] In response to the first operation, a second AI function control is displayed;

[0271] The receiving module is used for:

[0272] Receive selection operations for the second AI function control and text or voice commands input by the user;

[0273] The control module is used to control the first display element to execute the AI ​​function indicated by the text command or voice command.

[0274] In one possible implementation, the control module is used to control the display module:

[0275] Display the second AI function control combination, which includes multiple AI function controls. Different AI function controls in the second AI function control combination are the entry points for AI programs corresponding to different AI agents.

[0276] The receiving module is used for:

[0277] A selection operation is received for the second AI function control among the plurality of AI function controls.

[0278] In one possible implementation, the control module is used to control the display module:

[0279] Display the third AI function control or the collapse control;

[0280] The receiving module is configured to: receive operations on the third AI function control or the folding control;

[0281] The control module is used to unfold the third AI function control or the folded control to display the combination of the second AI function controls.

[0282] In one possible implementation, the first AI function control includes a content receiving area; the receiving module is specifically used for:

[0283] A drag operation is received to drag the first display element to the content receiving area.

[0284] In one possible implementation, the display position of the first AI function control is related to the position of the first display element.

[0285] In one possible implementation, the scenario includes time or location.

[0286] In one possible implementation, the control module is used to control the display module:

[0287] A dialog window is displayed, within which the first display element received by the AI ​​program is shown.

[0288] The following describes a terminal device provided in an embodiment of this application. The terminal device can be... Figure 13 For the data processing device in the middle, please refer to Figure 14 , Figure 14This is a schematic diagram of a terminal device provided in an embodiment of this application. The terminal device 800 can specifically be a virtual reality (VR) device, a mobile phone, a tablet, a laptop computer, a smart wearable device, etc., and is not limited thereto. Specifically, the terminal device 800 includes: a receiver 801, a transmitter 802, a processor 803, and a memory 804 (wherein the terminal device 800 may have one or more processors 803). Figure 8 (Taking a processor as an example), the processor 803 may include an application processor 8031 ​​and a communication processor 8032. In some embodiments of this application, the receiver 801, transmitter 802, processor 803, and memory 804 may be connected via a bus or other means.

[0289] Memory 804 may include read-only memory and random access memory, and provides instructions and data to processor 803. A portion of memory 804 may also include non-volatile random access memory (NVRAM). Memory 804 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0290] The processor 803 controls the operation of the terminal device. In specific applications, the various components of the terminal device are coupled together through a bus system. This bus system includes not only the data bus but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0291] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 803. Processor 803 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 803 or by instructions in software form. Processor 803 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 803 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 804. Processor 803 reads the information in memory 804 and, in conjunction with its hardware, completes the steps of the above method. Specifically, processor 803 can read the information in memory 804 and, in conjunction with its hardware, complete the data processing-related steps 301 to 303 in the above embodiments, as well as the data processing-related steps 301 to 303 in the above embodiments.

[0292] Receiver 801 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the terminal device. Transmitter 802 can be used to output digital or character information through the first interface; transmitter 802 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 802 may also include a display device such as a display screen.

[0293] This application also provides a computer program product that, when run on a computer, causes the computer to execute the above-described embodiments. Figure 3 The steps of the data processing method described in the corresponding embodiment.

[0294] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform the steps of the data processing method as described in the foregoing embodiments.

[0295] The data processing device provided in this application embodiment can specifically be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip in the execution device to execute the data processing method described in the above embodiment, or to cause the chip in the training device to execute the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit within the chip, such as a register or cache. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0296] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0297] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0298] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0299] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

Claims

1. A data processing method, characterized in that, Applied to a terminal device, the method includes: Receive the user's first action; In response to the first operation, a first display element on the display interface of the terminal device is selected and a first AI function control is displayed; wherein, the first AI function control is used to provide an input entry point for the AI ​​program; Upon receiving a preset operation for the first AI function control, the first display element is passed to the AI ​​program so that the AI ​​program can perform AI functions on the first display element.

2. The method according to claim 1, characterized in that, The method further includes: receiving text or voice commands input by the user; and passing the first display element to the AI ​​program so that the AI ​​program performs AI functions on the first display element, including: The first display element and the text or voice command are transmitted to the AI ​​program so that the AI ​​program can execute the AI ​​function indicated by the text or voice command on the first display element.

3. The method according to claim 1 or 2, characterized in that, The first operation is a selection operation of a first display element in the display interface of the terminal device; or, The first operation is to wake up the AI ​​program; or, The first operation is a selection operation on a blank area in the display interface.

4. The method according to any one of claims 1 to 3, characterized in that, The first displayed element is the object selected in the first operation; or, The first display element is determined based on the context information of the display interface of the object selected in the first operation; or, The first display element is selected from the display elements in the display interface based on at least one of the type of the display interface, the type of the display element in the display interface, the content of the display element in the display interface, and the display size of the display element.

5. The method according to any one of claims 1 to 4, characterized in that, The first AI function control for display includes: Display the first AI function control combination, which includes multiple AI function controls. Different AI function controls in the first AI function control combination are the entry points for AI programs corresponding to different AI agents. The received preset operation for the first AI function control includes: A preset operation is received for the first AI function control among the plurality of AI function controls.

6. The method according to claim 5, characterized in that, The combination of displaying the first AI function controls includes: Display the third AI function control or the collapse control; Receive an operation on the third AI function control or the folded control, and unfold the third AI function control or the folded control to display the first AI function control combination.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: In response to the first operation, a second AI function control is displayed; Upon receiving a selection operation for the second AI function control and a text or voice command input by the user, the AI ​​function indicated by the text or voice command is executed on the first display element.

8. The method according to claim 7, characterized in that, The second AI function control for display includes: Display the second AI function control combination, which includes multiple AI function controls. Different AI function controls in the second AI function control combination are the entry points for AI programs corresponding to different AI agents. The received selection operation for the second AI function control includes: A selection operation is received for the second AI function control among the plurality of AI function controls.

9. The method according to claim 8, characterized in that, The combination of displaying the second AI function controls includes: Display the third AI function control or the collapse control; Receive an operation on the third AI function control or the folded control, and unfold the third AI function control or the folded control to display the second AI function control combination.

10. The method according to any one of claims 1 to 9, characterized in that, The first AI function control includes a content receiving area; receiving a selection operation for the first AI function control includes: A drag operation is received to drag the first display element to the content receiving area.

11. The method according to any one of claims 1 to 10, characterized in that, The display position of the first AI function control is related to the position of the first display element.

12. The method according to any one of claims 1 to 11, characterized in that, The scenario includes time or location.

13. The method according to any one of claims 1 to 12, characterized in that, The method further includes: A dialog window is displayed, within which the first display element received by the AI ​​program is shown.

14. A data processing apparatus, characterized in that, Applied to a terminal device, the device includes: The receiving module is used to receive the user's first operation; The control module is used to respond to the first operation by controlling the display module to select a first display element on the display interface of the terminal device and display a first AI function control; wherein, the first AI function control is used to provide an input entry point for the AI ​​program; The receiving module is further configured to receive a selection operation for the first AI function control, and the control module is configured to pass the first display element to the AI ​​program so that the AI ​​program can perform AI functions on the first display element.

15. The apparatus according to claim 14, characterized in that, The receiving module is further configured to: receive text or voice commands input by the user; the control module is configured to: The first display element and the text or voice command are transmitted to the AI ​​program so that the AI ​​program can execute the AI ​​function indicated by the text or voice command on the first display element.

16. The apparatus according to claim 14 or 15, characterized in that, The first operation is a selection operation of a first display element in the display interface of the terminal device; or, The first operation is to wake up the AI ​​program; or, The first operation is a selection operation on a blank area in the display interface.

17. The apparatus according to any one of claims 14 to 16, characterized in that, The first displayed element is the object selected in the first operation; or, The first display element is determined based on the context information of the display interface of the object selected in the first operation; or, The first display element is selected from the display elements in the display interface based on at least one of the type of the display interface, the type of the display element in the display interface, the content of the display element in the display interface, and the display size of the display element.

18. The apparatus according to any one of claims 14 to 17, characterized in that, The control module is used to control the display module: Display the first AI function control combination, which includes multiple AI function controls. Different AI function controls in the first AI function control combination are the entry points for AI programs corresponding to different AI agents. The receiving module is used for: A preset operation is received for the first AI function control among the plurality of AI function controls.

19. The apparatus according to claim 18, characterized in that, The control module is used to control the display module to display the third AI function control or the folding control; The receiving module is used to receive operations on the third AI function control or the folding control, and the control module is used to control the display module to unfold and display the third AI function control or the folding control as the first AI function control combination.

20. The apparatus according to any one of claims 14 to 19, characterized in that, The control module is used to control the display module: In response to the first operation, a second AI function control is displayed; The receiving module is used for: Receive selection operations for the second AI function control and text or voice commands input by the user; The control module is used to control the first display element to execute the AI ​​function indicated by the text command or voice command.

21. The apparatus according to claim 20, characterized in that, The control module is used to control the display module: Display the second AI function control combination, which includes multiple AI function controls. Different AI function controls in the second AI function control combination are the entry points for AI programs corresponding to different AI agents. The receiving module is used for: A selection operation is received for the second AI function control among the plurality of AI function controls.

22. The apparatus according to claim 21, characterized in that, The control module is used to control the display module: Display the third AI function control or the collapse control; The receiving module is configured to: receive operations on the third AI function control or the folding control; The control module is used to unfold the third AI function control or the folded control to display the combination of the second AI function controls.

23. The apparatus according to any one of claims 14 to 22, characterized in that, The first AI function control includes a content receiving area; the receiving module is specifically used for: A drag operation is received to drag the first display element to the content receiving area.

24. The apparatus according to any one of claims 14 to 23, characterized in that, The display position of the first AI function control is related to the position of the first display element.

25. The apparatus according to any one of claims 14 to 24, characterized in that, The scenario includes time or location.

26. The apparatus according to any one of claims 14 to 25, characterized in that, The control module is used to control the display module: A dialog window is displayed, within which the first display element received by the AI ​​program is shown.

27. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the operation of the method according to any one of claims 1 to 13.

28. A computer program product, characterized in that, Includes computer-readable instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 13.

29. A system, characterized in that, It includes at least one processor and at least one memory; the processor and the memory are connected via a communication bus and communicate with each other. The at least one memory is used to store code; The at least one processor is used to execute the code to perform the method as described in any one of claims 1 to 13.

30. A chip, characterized in that, It includes at least one processing unit and an interface circuit, the interface circuit being used to provide program instructions or data to the at least one processing unit, the at least one processing unit being used to execute the program instructions to implement the method of any one of claims 1 to 13.

Images