A data processing method, electronic equipment and medium

Abstract

The application provides a data processing method, an electronic device and a medium, and relates to the technical field of terminals. The method can be applied to an electronic device, and comprises the following steps: the electronic device acquires a file descriptor of to-be-processed data, configures a pipeline for the to-be-processed data according to the file descriptor of the to-be-processed data, then adds a callback node in the pipeline, and configures a corresponding relationship between the callback node and a pre-constructed listener. The callback node is used for returning a data processing result output by the pipeline to the listener corresponding to the callback node. After the listener obtains the returned data, the listener can inform a target application of an application layer, and the target application obtains the returned data from the listener. In this way, in the data processing process, the target application does not need to continuously supervise the data processing result output by the pipeline, but the listener performs the supervision, and therefore, for the target application, it is a way of asynchronously obtaining the returned data, and other functions of the mobile phone used by a user are not affected.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of terminal, and in particular, to a data processing method, an electronic device and a medium. BACKGROUND

[0002] With the development of terminal technology, electronic devices such as mobile phones also develop rapidly, and the storage capacity of mobile phones is gradually improved. Generally, a large amount of multimedia data such as pictures, videos and audios will be stored on a mobile phone.

[0003] Generally, a mobile phone can process the above multimedia data, for example, a user can edit the multimedia data through the mobile phone. The data size of pictures and audios is usually smaller than that of videos, and the mobile phone processes data with a smaller data size (such as pictures) faster, and processes data with a larger data size (such as videos) slower, which consumes more time.

[0004] However, during the process of processing data with a larger data size by the mobile phone, the user cannot use other functions of the mobile phone during data processing, which affects the user experience. SUMMARY

[0005] The present application provides a data processing method, an electronic device and a medium, which can enable the user to use other functions of the mobile phone during data processing, and improve the user experience.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a data processing method applied to an electronic device, which can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a notebook computer, a handheld computer, a netbook, a wearable electronic device, a smart watch and the like. Specifically, the method comprises: the electronic device acquires a file descriptor of to-be-processed data, configures a pipeline for the to-be-processed data according to the file descriptor of the to-be-processed data, then adds a callback node in the pipeline, and configures a corresponding relationship between the callback node and a pre-constructed listener, the callback node is used to return the data processing result output by the pipeline to the listener corresponding to the callback node, and after the listener obtains the returned data, the listener can inform a target application of an application layer, and the target application obtains the returned data from the listener. In this way, during data processing, the target application does not need to continuously supervise the data processing result output by the pipeline, but the listener performs the supervision, and it can be seen that for the target application, it is an asynchronous way of obtaining the returned data, which does not affect the user to use other functions of the mobile phone.

[0008] In some possible implementation manners, the electronic device can add the callback node in the pipeline when the data stream of the to-be-processed data is greater than a preset threshold.

[0009] In some possible implementation manners, the electronic device can add a callback node after the last node of the pipeline. In this way, the data processing result of the pipeline after processing the data is given to the callback node for data return.

[0010] In some possible implementation manners, the electronic device can parse the file descriptor of the to-be-processed data to obtain the processing node for the to-be-processed data and the connection order of the processing node; and connect the processing nodes according to the connection order of the processing node to obtain the pipeline for the to-be-processed data.

[0011] In some possible implementation manners, the electronic device can display a user interface of the target application, which can include a target control, for example, a one-key-to-clip control, used to instruct to process the to-be-processed data. The electronic device can obtain the file descriptor of the to-be-processed data in response to a triggering operation on the target control.

[0012] In some possible implementation manners, the electronic device includes an application layer and a media middleware post-processing framework layer, the application layer includes a target application, and the media middleware post-processing framework layer includes a solution layer and a core framework layer; adding a callback node in the pipeline and configuring a corresponding relationship between the callback node and a pre-constructed listener includes: the target application sends a configuration listener instruction to the core framework layer; the core framework layer configures the listener based on the configuration listener instruction; the target application sends a registration callback instruction to the solution layer; the solution layer sends a configuration callback instruction to the core framework layer based on the registration callback instruction; and the core framework layer configures the corresponding relationship between the callback node and the listener based on the configuration callback instruction.

[0013] In some possible implementation manners, obtaining the return data from the listener includes: after the listener receives the return data of the callback node, the listener reports a notification to the target application of the application layer; and the target application obtains the return data from the listener based on the notification reported by the listener.

[0014] In some possible implementation manners, the core framework layer includes a node management module, a data listening module, and a pipeline module; and configuring the pipeline for the to-be-processed data according to the file descriptor of the to-be-processed data includes: the solution layer sends a node configuration instruction to the node management module; the node configuration module configures the node for the to-be-processed data based on the node configuration instruction, and sends a first connection instruction to the pipeline module; and the pipeline module connects the node for the to-be-processed data into the pipeline based on the first connection instruction.

[0015] In a second aspect, the present application provides an electronic device, comprising a memory and a processor; the memory is configured to store a computer program; the processor is configured to execute the computer program in the memory to implement the method in any possible implementation manner of the first aspect.

[0016] In a third aspect, the present application provides a computer readable storage medium, having stored thereon a computer program, which when executed by a processor implements the method in any possible implementation manner of the first aspect.

[0017] In a fourth aspect, the present application provides a computer program product, which when executed on a computer causes the computer to perform the method in any possible implementation manner of the first aspect.

[0018] The technical scheme of the present application has the following beneficial effects:

[0019] The present application provides a data processing method, which comprises: an electronic device acquires a file descriptor of to-be-processed data, configures a pipeline for the to-be-processed data according to the file descriptor of the to-be-processed data, then adds a callback node in the pipeline, and configures a corresponding relationship between the callback node and a pre-constructed listener, the callback node being configured to return a data processing result output by the pipeline to the listener corresponding to the callback node, the listener can inform a target application of an application layer after obtaining the returned data, and the target application obtains the returned data from the listener. In this way, in the data processing process, the target application does not need to continuously supervise the data processing result output by the pipeline, but the listener performs the supervision, and therefore, for the target application, it is an asynchronous manner of obtaining the returned data, and other functions of the mobile phone are not affected. In the method, the user can use other functions of the mobile phone during the data processing, and the experience of the user is improved. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 A composition schematic diagram of an electronic device provided by an embodiment of the present application is shown in the figure;

[0021] Figure 2 A software structure schematic diagram of an electronic device provided by an embodiment of the present application is shown in the figure;

[0022] Figure 3 A flowchart of a data processing method provided by an embodiment of the present application is shown in the figure;

[0023] Figure 4 An opening gallery application schematic diagram provided by an embodiment of the present application is shown in the figure;

[0024] Figure 5A A gallery interface schematic diagram provided by an embodiment of the present application is shown in the figure;

[0025] Figure 5B A schematic diagram of a photo album interface provided by an embodiment of the present application;

[0026] Figure 5C A schematic diagram of a display picture interface provided by an embodiment of the present application;

[0027] Figure 5D A schematic diagram of an editing interface provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] The terms "first", "second", and "third" and the like in the description and in the claims of the present application and the summary of the application, unless otherwise pridicating a specific sequence, refer to different objects and not to a particular order.

[0029] In the present application embodiments, the words "exemplary" and "for example" are used to mean "an example of" or "an example, only. Any implementation or design solution described herein as "exemplary" or "for example" is not necessarily to be construed as preferred or advantageous over other implementations or design solutions. In fact, the term "exemplary" or "for example" is used to present concepts in a concrete manner.

[0030] For the sake of clear and concise description of the following embodiments, first, a brief introduction of the related art is given:

[0031] The media middleware post-processing framework refers to a framework integrating multiple media data processing functions. Different functions can be embodied by different nodes, that is, different nodes embody different functions. In some examples, different electronic devices have different node capabilities of the media middleware post-processing framework. An application in the application layer can query the supported capabilities of the electronic device through the media middleware post-processing framework, and call the related capabilities, thereby implementing various functions, such as video editing, image shooting, and the like. In some examples, the electronic device can dynamically display function controls, and thus different users experience different functions for different electronic devices.

[0032] Generally, in order to facilitate user use, after a mobile phone is shipped, the mobile phone is equipped with some built-in applications, such as a camera, a gallery, a phone, a message, a browser, and a universal remote control. In order to arouse user interest and improve the popularity of the mobile phone, a function of editing pictures and / or videos is designed in the above-mentioned gallery application. For example, a one-key film making function can be designed in the photo album application. Based on the one-key film making function, a user only needs to select materials (pictures and / or videos), and then based on the one-key film making control, the mobile phone will automatically edit the materials, and thus obtain processed multimedia data.

[0033] However, during the editing process of the mobile phone on the material, the user cannot use other functions of the mobile phone during the processing, the user needs to wait until the editing process is completed, and then the user can use other functions of the mobile phone, and when the data of the material is large, the time required for the mobile phone to edit the process is long, which increases the waiting time of the user and affects the user experience.

[0034] Therefore, the embodiment of the present application provides a data processing method, which can be applied to an electronic device. During the processing of data, the electronic device can first determine the data size of the data to be processed. If the data size of the data to be processed is less than a preset threshold, a synchronous processing mode is selected for processing of the data with small data size (for example, a picture). If the data size of the data to be processed is greater than or equal to the preset threshold, an asynchronous processing mode is selected for processing of the data with large data size (for example, a video). In the asynchronous processing mode, taking the processing of video data by the electronic device as an example, the electronic device only needs to register a callback node after a video processing node. Thus, during the processing of the video by the video processing node, the callback node actively reports the video data processing result after obtaining the video data processing result. The user can use other functions of the mobile phone during the waiting process of the mobile phone for processing of the data, thereby improving the user experience.

[0035] In some embodiments, the electronic device can also be a tablet computer, a desktop computer, a laptop computer, a notebook computer, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a personal digital assistant (PDA), a wearable electronic device, a smart watch, and the like. The specific form of the electronic device is not specially limited in the present application. In the present embodiment, the structure of the electronic device can be as shown in Figure 1 Figure 1 which is a structural schematic diagram of an electronic device provided in the present application.

[0036] As shown in Figure 1 , the electronic device can include a processor 110, an internal memory 121, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, a sensor module 180, a display screen 194, and the like. The sensor module 180 can include a touch sensor 180K and the like.

[0037] It can be understood that the structure shown in the present embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, a combination of software and hardware. ​

[0038] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors. For example, in the present application, the processor 110 can obtain a file descriptor of to-be-processed data; configure a pipeline for the to-be-processed data according to the file descriptor of the to-be-processed data; add a callback node in the pipeline, and configure a corresponding relationship between the callback node and a pre-constructed listener, the callback node being configured to return a data processing result output by the pipeline to the listener corresponding to the callback node; and obtain the returned data from the listener.

[0039] Among them, the controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to instruction operation codes and timing signals to complete the control of fetching and executing instructions.

[0040] The memory in the processor 110 can also be configured to store instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can save instructions or data that have just been used or are recycled by the processor 110. If the processor 110 needs to use the instructions or data again, it can directly call from the memory. Avoiding repeated access, reducing the waiting time of the processor 110, thus improving the efficiency of the system.

[0041] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0042] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 can include multiple sets of I2C buses. The processor 110 can be coupled to the touch sensor 180K, etc. through different I2C bus interfaces, respectively. For example, the processor 110 can be coupled to the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface to implement the touch function of the electronic device. Based on the touch function of the electronic device, after a user touches an application icon displayed by the electronic device, the electronic device can start the application.

[0043] The UART interface is a general serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is usually used to connect the processor 110 and the wireless communication module 160.

[0044] The MIPI interface can be used to connect the processor 110 and peripheral devices such as the display screen 194. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), and the like. In some embodiments, the processor 110 and the camera communicate through the CSI interface to implement the photographing function of the electronic device. The processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the electronic device. For example, the electronic device can display relevant interfaces based on the display function.

[0045] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 and the camera, the display screen 194, the wireless communication module 160, the sensor module 180, and the like. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, and the like.

[0046] It can be understood that the interface connection relationship between the modules illustrated in the embodiments is only illustrative and does not constitute a structural limitation on the electronic device. In some other embodiments of the present application, the electronic device can also use different interface connection methods or a combination of multiple interface connection methods.

[0047] The wireless communication function of the electronic device can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, and the like.

[0048] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0049] The mobile communication module 150 can provide a solution for wireless communication including 2G / 3G / 4G / 5G, etc. applied to the electronic device. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive an electromagnetic wave by the antenna 1, and perform filtering, amplification, etc. on the received electromagnetic wave, and transfer the processed signal to the modem processor for demodulation. The mobile communication module 150 can also amplify a signal modulated by the modem processor, and radiate the amplified signal as an electromagnetic wave through the antenna 1. In some embodiments, at least part of the function modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least part of the function modules of the mobile communication module 150 can be disposed in the same device as at least part of the modules of the processor 110.

[0050] The modem processor can include a modulator and a demodulator. The modulator is configured to modulate a low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is configured to demodulate a received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal processed by the baseband processor is transmitted to the application processor. The application processor displays images or videos through the display screen 194. In some embodiments, the modem processor can be a separate device. In other embodiments, the modem processor can be independent of the processor 110, and disposed in the same device as the mobile communication module 150 or other function modules.

[0051] The wireless communication module 160 can provide a solution for wireless communication including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. applied to the electronic device. The wireless communication module 160 can be one or more devices integrated with at least one communication processing module. The wireless communication module 160 receives an electromagnetic wave via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signal, and transmits the processed signal to the processor 110. The wireless communication module 160 can also receive a signal to be transmitted from the processor 110, perform frequency modulation and amplification, and radiate the processed signal as an electromagnetic wave through the antenna 2.

[0052] In some embodiments, the antennas 1 and the mobile communication module 150 are coupled, and the antennas 2 and the wireless communication module 160 are coupled, so that the electronic device can communicate with a network and other devices through wireless communication technology. The wireless communication technology can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include a global positioning system (GPS), a global navigation satellite system (GLONASS), a beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS), and / or a satellite based augmentation systems (SBAS).

[0053] The electronic device implements a display function through a GPU, a display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, which is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs, which execute program instructions to generate or change display information.

[0054] The display screen 194 is configured to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oled, a quantum dot light emitting diodes (QLED), or the like. In some embodiments, the electronic device can include one or N display screens 194, where N is a positive integer greater than 1.

[0055] A series of graphical user interfaces (GUIs) can be displayed on the display screen 194 of the electronic device, and these GUIs are home screens of the electronic device. Generally, the size of the display screen 194 of the electronic device is fixed, and only limited controls can be displayed in the display screen 194 of the electronic device. A control is a GUI element, which is a software component included in an application program, controls all data processed by the application program and the interactive operations related to the data, and a user can interact with the control through direct manipulation to read or edit the relevant information of the application program. Generally, a control can include an icon, a button, a menu, a tab, a text box, a dialog box, a status bar, a navigation bar, a widget, and the like visual interface elements.

[0056] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in internal memory 121. For example, in this embodiment, processor 110 can obtain a file descriptor of data to be processed by executing instructions stored in internal memory 121; configure a pipeline for the data to be processed according to the file descriptor of the data to be processed; add callback nodes to the pipeline and configure the correspondence between the callback nodes and pre-built listeners, wherein the callback nodes are used to send the data processing results output by the pipeline back to the listener corresponding to the callback node; and obtain the returned data from the listener. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, phone book, etc.), etc. In addition, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. The processor 110 performs various functional applications and data processing of the electronic device by executing instructions stored in the internal memory 121 and / or instructions stored in memory disposed in the processor.

[0057] In addition, an operating system runs on top of these components. Examples include Apple's iOS, Google's Android, and Microsoft's Windows. Applications can be installed and run on this operating system.

[0058] The operating system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses the layered architecture Android system as an example to illustrate the software structure of an electronic device.

[0059] like Figure 2 As shown, this figure is a schematic diagram of the software structure of an electronic device provided in an embodiment of this application.

[0060] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is layered from top to bottom as follows: application layer, media middleware post-processing framework layer, application framework layer, hardware abstraction layer, and kernel layer.

[0061] The application layer can include a series of applications, such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, and the like. In the embodiment of the present application, the video clip function is included in the gallery application, which can be automatic clipping or manual clipping.

[0062] The media middleware post-processing framework layer can include a solution layer and a core framework layer. The media middleware post-processing framework layer can be located between the application layer and the application program framework layer.

[0063] The solution layer includes an interface module, which is used to manage a video analysis interface, a picture analysis interface, a theme analysis interface, and the like. Different interfaces are used to implement different functions. For example, the video analysis interface is used to implement processing of a video, the picture analysis interface is used to implement processing of a picture, and the theme analysis interface is used to implement analysis of a theme.

[0064] The core framework layer includes a pipeline module, a node management module, a data monitoring module, and a plurality of data processing nodes and callback nodes. In the embodiment of the present application, taking one-key-to-clip as an example, the plurality of data processing nodes corresponding to the one-key-to-clip function can include a processing node for video data and a processing node for picture data. The processing node for video data can include a file analysis node, a video decoding node, and a frame reduction node, and the processing node for picture data can include a picture analysis node.

[0065] The pipeline module is used to connect the plurality of data processing nodes and the callback nodes to obtain a pipeline connected by the nodes. When the pipeline runs, the data to be processed flows from an initial node of the pipeline to a terminal node of the pipeline based on the node order in the pipeline. The initial node processes or processes the data to be processed, and the subsequent node further processes the output node of the previous node until the terminal node is processed.

[0066] The node management module is used to manage the nodes, for example, can include generating nodes. The node can be a function, the node can be pre-set, the node management module generates the node, and the node management module can load the pre-set node, such as loading the resource of the node.

[0067] The data monitoring module is used to monitor whether the callback node established by the node management module has returned data. The callback node can return the data to the node management module after receiving the data. The node management module receives the data returned by the callback node, and transmits the data to the data monitoring module. The data monitoring module can notify the application of the application layer to retrieve the data returned by the callback node.

[0068] The file parsing node is used to output a data stream, which can be an audio stream, a video stream, or an audio stream and a video stream. In some examples, the file parsing node obtains meta information in a file through a MediaExtractor parser, and passes the meta information to a next node, such as a video decoding node. The input of the file parsing node can be a file descriptor (fd) or a file path, and the output of the file parsing node can be video compression data, meta information, a Multipurpose Internet Mail Extension (MIME) type, and the like.

[0069] The video decoding node is used to receive a MIME type and meta information, construct an audio decoder and a video decoder, decode multimedia data stream through a codec component (such as a MediaCodec), and pass the decoded data to a next node, such as a frame reduction node. The input of the video decoding node can be meta information and a MIME type of a file, and the output is a multimedia data stream (such as a video stream or an audio stream).

[0070] The frame reduction node is used to selectively discard video frame data in a video stream through a frame dropping algorithm or a frame reduction algorithm, for example, to reduce 60fps data to 30fps. The input of the frame reduction node can be video frame data, and the output is also video frame data.

[0071] The picture analysis node is used to analyze picture data frames inputted, and output analysis results, such as specific content, scenes, characters, and the like in a picture. Subsequent related modules and nodes are described in detail.

[0072] The application framework layer provides an application programming interface (API) and a programming framework for an application in the application layer. The application framework layer includes some pre-defined functions. The application framework layer can include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0073] The window manager is used to manage a window program. The window manager can obtain a display screen size, determine whether there is a status bar, lock a screen, and capture a screen, and the like.

[0074] The content provider is used to store and obtain data, and make the data accessible to an application. The data can include videos, images, audios, dialed and received calls, browsing history and bookmarks, a phone book, and the like.

[0075] The view system includes visual controls, such as controls that display text, controls that display pictures, and the like. The view system can be used to build an application. A display interface can be composed of one or more views. For example, a display interface that includes a short message notification icon can include a view that displays text and a view that displays a picture. In some examples, the view system can display corresponding functional controls based on the nodes supported by the phone itself.

[0076] The phone manager is used to provide communication functions of the electronic device. For example, management of call status (including call connection, call hang-up, and the like).

[0077] The resource manager provides various resources for an application, such as localized strings, icons, pictures, layout files, video files, and the like.

[0078] The notification manager enables an application to display notification information in a status bar, which can be used to convey a message of the notification type, which can automatically disappear after a short period of time without user interaction. For example, the notification manager is used to notify of a download completion, a message reminder, and the like. The notification manager can also be a notification that appears in a top status bar of the system in the form of a graph or a scroll bar text, a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is prompted in a status bar, a prompt sound is emitted, the electronic device is vibrated, an indicator light is blinked, and the like.

[0079] The application layer and the application framework layer are run in a virtual machine. The virtual machine executes java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions of management of an object life cycle, stack management, thread management, security and exception management, and garbage collection.

[0080] The system library can include a plurality of functional modules. For example, a surface manager, media libraries, a three-dimensional graphics processing library (for example, OpenGL ES), a 2D graphics engine (for example, SGL), and the like.

[0081] The surface manager is used to manage a display subsystem and to provide fusion of 2D and 3D layers for a plurality of applications.

[0082] The media library supports playback and recording of a plurality of commonly used audio, video formats, and still image files. The media library can support a plurality of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, and the like.

[0083] The three-dimensional graphics processing library is used to implement three-dimensional graphics drawing, image rendering, composition, and layer processing.

[0084] The 2D graphics engine is a drawing engine for 2D drawing.

[0085] The kernel layer is a layer between hardware and software. The kernel layer can include display drivers, camera drivers, audio drivers, sensor drivers, and the like.

[0086] It should be noted that although the embodiments of the present application are described by taking the Android system as an example, the basic principles are also applicable to electronic devices based on iOS, Windows, and the like operating systems.

[0087] In order to make the technical solutions of the present application clearer and easier to understand, the following will be combined with Figure 2 and Figure 3 introduce a data processing method provided by the embodiments of the present application. As Figure 3 described, the figure is a flowchart of a data processing method provided by the embodiments of the present application, and the method comprises:

[0088] S301, the electronic device starts a target application.

[0089] The target application can be any one of the applications in the application layer, for example, can be a gallery application, or can be a camera application. In order to facilitate understanding, the technical solutions of the present application will be introduced by taking the electronic device as a mobile phone and the target application as a gallery application.

[0090] In some examples, the electronic device can receive a touch operation of a user on an icon of the target application in the home interface, and start the target application. As Figure 4 shown, the figure is a schematic diagram of opening a gallery application provided by the embodiments of the present application. The touch sensor 180K of the mobile phone can receive a touch operation of a user on an icon of the target application in the home interface of the mobile phone, and report to the processor 110, and the processor 110 can display a user interface of the target application in the display screen 194 in response to the touch operation. In addition, the mobile phone can also be started by other ways in the embodiments of the present application, and the user interface of the gallery application is displayed on the display screen 194. For example, when the mobile phone is in a black screen, a lock screen, or a user interface of a certain application, the processor 110 can start the gallery application and display the user interface of the gallery application in the display screen 194 in response to a voice instruction or a shortcut operation of the user, and the like.

[0091] In order to facilitate understanding, the following will be introduced by taking the data processing process of the electronic device when the user uses the one-key photo function of the gallery application as an example.

[0092] S302, the electronic device receives a trigger operation of a user on a one-key photo control.

[0093] The triggering operation can be a click, a long press, or the like. After the target application is started, the electronic device can present a relevant interface of the target application, such as the gallery interface 410 shown in FIG. 4A, which is a schematic diagram of a gallery interface provided by an embodiment of the present application. The gallery interface includes multiple albums, such as album 410, album 420, and album 430. A user can click the album 410, and after the electronic device receives the click operation of the user on the album 410, the electronic device can display the photos or videos included in the album 410. Figure 5A

[0094] As shown in FIG. 4B, which is a schematic diagram of an album interface provided by an embodiment of the present application, the album interface includes multiple photos, such as photo 411, photo 412, and photo 413. A user can select a photo by clicking the photo. For example, the user can click the photo 411, and the electronic device can display the photo 411 in a zoomed-in manner and display a one-click photo creation control. Figure 5B

[0095] As shown in FIG. 4C, which is a schematic diagram of a display photo interface provided by an embodiment of the present application, the display photo interface includes a one-click photo creation control 440. After a user clicks the one-click photo creation control 440, the electronic device receives the triggering operation of the user on the one-click photo creation control and displays an editing interface for the photo. The user interface of the target application can be an editing interface of the gallery, and the target control can be the one-click photo creation control 440. Figure 5C

[0096] As shown in FIG. 4D, which is a schematic diagram of an editing interface provided by an embodiment of the present application, the electronic device can display, in the editing interface, the selected materials, the photo creation mode, and the like. A user can also reselect the materials in the editing interface. Figure 5D

[0097] S303. The target application sends the fd of the user-selected materials to the solution layer.

[0098] In some examples, the materials can include pictures and / or videos. The solution layer maintains an interface module for managing a video analysis interface, a picture analysis interface, a theme analysis interface, and the like. When the user-selected materials are pictures, the target application can pass the fd of the pictures to the picture analysis interface. When the user-selected materials are videos, the target application can pass the fd of the videos to the video analysis interface. Of course, if the user-selected materials include both pictures and videos, the target application can send the fd of the pictures to the picture analysis interface and the fd of the videos to the video analysis interface, respectively.

[0099] It should be noted that the above merely takes the fd of the materials as an example. In some examples, the target application also sends the identity, path, and the like of the materials to the interface module.

[0100] ​​​​S304. The solution layer sends a node configuration instruction to the node management module.

[0101] The node configuration instruction is an instruction for instructing the node management module to configure a node. After the interface module receives the fd of the material passed by the target application, the solution layer can instruct the node management module to configure the corresponding node. The target application can be a gallery, and the gallery has a clipping function, such as a one-key film function. The node configuration instruction can carry configuration information corresponding to the one-key film function, which is used to instruct which nodes need to be configured. For example, the configuration information can be an instruction to configure a processing node for video data and a processing node for picture data.

[0102] S305. The node management module configures the corresponding node based on the node configuration instruction.

[0103] As described above, the node configuration instruction is used to instruct the node management module to configure the corresponding node. Continuing the above example, after receiving the node configuration instruction, the node management module can configure the processing node for video data and the processing node for picture data based on the configuration information carried in the node configuration instruction. For example, for video data, the node management module can configure a file parsing node, a video decoding node, and a frame reduction node, and for picture data, the node management module can configure a picture analysis node.

[0104] It should be noted that the file parsing node, the video decoding node, the frame reduction node, and the picture analysis node can be pre-constructed, and the configuration process of the node management module for the file parsing node, the video decoding node, the frame reduction node, and the picture analysis node can be to load the code or resource library of the node. For example, the process of the node management module configuring the file parsing node can be to load the code or resource library of the file parsing node.

[0105] S306. The node management module sends a first connection instruction to the data pipeline module.

[0106] The first connection instruction is an instruction for instructing the nodes to be connected in a certain order, and the first connection instruction carries the connection order of each node. For example, the first connection instruction carries the connection order of the file parsing node, the connection order of the video decoding node, the connection order of the frame reduction node, and the connection order of the picture analysis node. The connection order of each node can be pre-set, or can be obtained by the target application by adaptively adjusting the pre-set order based on the function to be implemented.

[0107] S307. The pipeline module connects the nodes into a pipeline based on the first connection instruction.

[0108] The pipeline refers to a node line formed by at least one node connected in a certain order. The order of the nodes in the pipeline indicates the order in which the nodes process data, and the nodes in the pipeline process data in turn to obtain processed data.

[0109] After the pipeline module receives the first connection instruction, the pipeline module can connect the file parsing node, the video decoding node and the frame reduction node into a video processing pipeline based on the connection order carried in the first connection instruction. At this time, the order of the nodes in the video processing pipeline is file parsing node, video decoding node and frame reduction node in turn. The pipeline module can also connect the picture analysis node into a picture processing pipeline based on the connection order carried in the first connection instruction. The order of the nodes in the picture processing pipeline is picture analysis node in turn.

[0110] S308, the pipeline module returns a connection completion instruction to the target application.

[0111] The connection completion instruction is used to indicate that the pipeline module has completed the connection of the nodes. For example, the pipeline module connects the file parsing node, the video decoding node and the frame reduction node into a video processing pipeline, and connects the picture analysis node into a picture processing pipeline.

[0112] S309, the target application sends a configuration listener instruction to the data listener module.

[0113] The configuration listener instruction is an instruction used to indicate the configuration of the listener. After the target application receives the connection completion instruction returned by the pipeline module, the target application configures the listener and the callback node to realize the listening and return of the processed data.

[0114] S310, the data listener module configures the listener based on the configuration listener instruction.

[0115] After the data listener module receives the configuration listener instruction, the data listener module configures the listener based on the configuration listener instruction.

[0116] S311, the data listener module returns a listener configuration completion instruction to the target application.

[0117] After the data listener module completes the configuration of the listener, the data listener module returns the listener configuration completion instruction to the target application to inform the target application that the configuration of the listener has been completed.

[0118] S312, the target application sends a registration callback instruction to the solution layer.

[0119] The registration callback instruction is an instruction for instructing registration of a callback node, where the callback node is a node for receiving processed data, and the callback node transmits the processed data to the node management module. After the data monitoring module completes configuration of the listener, the target application sends a registration callback instruction to the solution layer to register the callback node.

[0120] In S313, the solution layer queries the callback node based on the registration callback instruction.

[0121] The callback node can be pre-built, and the solution layer can query the pre-built callback node after receiving the registration callback instruction. If the pre-built callback node is not queried, the registration fails. If the pre-built callback node is queried, the pre-built callback node is registered, that is, the newly configured listener is associated with the callback node. In this way, the node management module knows that the listener exists, and after the callback node transmits data to the node management module, the node management module forwards the data to the listener. After the listener receives the data, the listener can report to the target application that new data is generated and notify the target application to get the new data.

[0122] In S314, the solution layer sends a configuration callback instruction to the node management module.

[0123] The configuration callback instruction is an instruction for instructing configuration of the callback node. After the solution layer finds the callback node, the solution layer can send the configuration callback instruction to the node management module to configure the callback node.

[0124] In S315, the node management module configures the callback node based on the configuration callback instruction.

[0125] The node management module can load the code or resource library of the callback node. The node management module can also configure the correspondence between the callback node and the listener, and then maintain the correspondence in the node management module, so that the node management module can process or forward the data transmitted by the callback node in the future.

[0126] In S316, the node management module sends a second connection instruction to the pipeline module.

[0127] The second connection instruction is an instruction for adding the callback node to the pipeline. The second connection instruction can include a connection position of the callback node, and the connection position can be after the frame reduction node in the video processing pipeline. The node management module can send the second connection instruction to the pipeline module to add the callback node to the video processing pipeline.

[0128] S317, the pipeline module connects the callback node in the pipeline based on the second connection instruction.

[0129] After the pipeline module receives the second connection instruction, the callback node can be connected in the video processing pipeline based on the connection position of the callback node carried in the second connection instruction.

[0130] S318, the pipeline module returns a connection completion instruction to the target application.

[0131] After the pipeline module completes the addition of the callback node, a connection completion instruction is returned to the target application, so that the target application knows that the connection of the callback node has been completed.

[0132] S319, the target application transmits a running instruction to the pipeline module.

[0133] The running instruction is an instruction for instructing the pipeline module to run the pipeline. After the pipeline module completes the addition of the callback node, the target application sends a running instruction to the pipeline module.

[0134] S320, the pipeline module runs the pipeline based on the running instruction.

[0135] The pipeline can include a video processing pipeline and a picture processing pipeline, which can be referred to Figure 2 . The video processing pipeline includes a file parsing node, a video decoding node, and a frame reduction node connected in order, and the callback node is connected at the end of the video processing pipeline, i.e., after the frame reduction node.

[0136] After the video processing pipeline starts running, the file parsing node can receive the fd of the file or the path of the file transmitted by the target application through the video analysis interface, etc. Taking the fd of the file transmitted through the video analysis interface as an example, the file parsing node determines the file based on the fd of the file, then obtains the meta information in the file through the MediaExtractor parser, and passes the meta information to the next node, such as the video decoding node. The output of the file parsing node can also be video compression data, MIME type, etc.

[0137] The video decoding node can receive the MIME type and meta information output by the file parsing node, build an audio decoder and a video decoder, decode the multimedia data stream through the MediaCodec component, and transmit the decoded data to the next node, such as the frame reduction node, for example, the multimedia data stream (audio stream, video stream) can be transmitted to the frame reduction node.

[0138] The frame reduction node receives the multimedia data stream sent by the video decoding node, and can selectively discard the video frame data in the video stream through a frame dropping algorithm or a frame reduction algorithm to achieve frame reduction processing, and then transmit the video frame data after frame dropping processing to the callback node.

[0139] After receiving the video frame data after frame dropping processing, the callback node can transmit the data to the node management module to inform the node management module that there is currently return data, i.e., video frame data after frame reduction processing.

[0140] The picture processing pipeline includes a picture analysis node configured to analyze the picture data frame and obtain an analysis result. For example, an image processing algorithm can be used to determine information such as the specific content, scene, and characters of the picture.

[0141] S321, the node management module sends the return data to the data monitoring module.

[0142] The data monitoring module maintains a listener. After the listener and the callback node complete registration, the node management module maintains a correspondence between the callback node and the listener. When the node management module receives the return data of the callback node, the return data is sent to the listener corresponding to the callback node based on the correspondence. For example, callback node 1 corresponds to listener 1, and callback node 2 corresponds to listener 2. When the node management module receives the return data returned by callback node 1, the return data is sent to listener 1.

[0143] S322, the data monitoring module notifies the target application to retrieve the return data.

[0144] After the data monitoring module receives the data returned by the node management module, it reports to the application layer to inform the target application to retrieve the return data. As can be seen, in the present application, during the process of processing data by the mobile phone, the listener and the callback node are set. The callback node is used to receive the data processed by the video processing pipeline, and the listener can monitor the return data of the callback node. That is, once there is return data, the listener can automatically report to the application layer, and the target application of the application layer can automatically retrieve the return data from the listener. In this process, the user can use other functions of the mobile phone without waiting for the mobile phone to process the data, thereby improving the user experience.

[0145] S323, the target application obtains the return data from the data monitoring module.

[0146] After receiving the report from the data monitoring module, the target application obtains the return data from the data monitoring module.

[0147] S324, the data monitoring module returns a receiving completion instruction to the node management module.

[0148] The receive completion instruction is used to indicate the completion of data reception. After the data monitoring module completes the reception of the data sent back by the node management module, it sends a receive completion instruction to the node management module.

[0149] S325, The node management module sends a callback completion instruction to the pipeline module.

[0150] A callback completion instruction is an instruction used to indicate that the current data callback has been completed. After receiving the reception completion instruction, the node management module sends a callback completion instruction to the pipeline module to inform the pipeline module that the current data callback has been completed, that is, the current data processing is finished.

[0151] Based on the above description, this application provides a data processing method that can be applied to an electronic device. Before processing data, the electronic device sets up a listener and a callback node. The callback node receives the processed data, and the listener monitors the processed data. Therefore, in this application, the listener, acting on behalf of the "user," monitors whether data processing is complete. If data processing is complete, the callback node sends data back to the listener, which then reports to the application layer. The target application in the application layer retrieves the returned data from the listener. Thus, while the phone is processing data, the user can still use other functions of the phone, improving the user experience.

[0152] This embodiment also provides a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform... Figure 3 The relevant methods and steps are described in the above embodiments to implement the methods described.

[0153] This embodiment also provides a computer program product containing instructions that, when run on an electronic device, cause the electronic device to perform actions such as... Figure 3 The relevant methods and steps are described in the above embodiments to implement the methods described.

Claims

1. A data processing method, characterized by, The method is applied to an electronic device, and comprises: obtaining a file descriptor of to-be-processed data; configuring a pipeline for the to-be-processed data according to the file descriptor of the to-be-processed data; adding a callback node after a last node of the pipeline, and configuring a correspondence between the callback node and a pre-constructed listener, the callback node being configured to return a data processing result output by the pipeline to the listener corresponding to the callback node; the callback node refers to a node configured to receive data after processing is completed; obtaining the returned data from the listener.

2. The method of claim 1, wherein, The adding of the callback node in the pipeline comprises: if a data volume of the to-be-processed data is greater than a preset threshold, adding the callback node in the pipeline.

3. The method of claim 1, wherein, The configuring of the pipeline for the to-be-processed data according to the file descriptor of the to-be-processed data comprises: parsing the file descriptor of the to-be-processed data to obtain processing nodes for the to-be-processed data and a connection order of the processing nodes; connecting the processing nodes according to the connection order of the processing nodes to obtain the pipeline for the to-be-processed data.

4. The method according to any one of claims 1 to 3, characterized in that, Before the obtaining of the file descriptor of the to-be-processed data, the method further comprises: displaying a user interface of a target application, the user interface comprising a target control configured to instruct processing of to-be-processed data; The obtaining of the file descriptor of the to-be-processed data comprises: in response to a triggering operation on the target control, obtaining the file descriptor of the to-be-processed data.

5. The method according to any one of claims 1 to 4, characterized in that, The electronic device comprises an application layer and a media middleware post-processing framework layer, the application layer comprising a target application, and the media middleware post-processing framework layer comprising a solution layer and a core framework layer; The adding of the callback node in the pipeline and the configuring of the correspondence between the callback node and the pre-constructed listener comprise: the target application sending a configuration listener instruction to the core framework layer; the core framework layer configuring a listener based on the configuration listener instruction; the target application sending a registration callback instruction to the solution layer; the solution layer sending a configuration callback instruction to the core framework layer based on the registration callback instruction; the core framework layer configuring the correspondence between the callback node and the listener based on the configuration callback instruction.

6. The method of claim 5, wherein, The obtaining of the returned data from the listener comprises: after the listener receives the returned data of the callback node, the listener reporting a notification to the target application of the application layer; the target application obtaining the returned data from the listener based on the notification reported by the listener.

7. The method of claim 6, wherein, The core framework layer comprises a node management module, a data listening module and a pipeline module; the configuring of the pipeline for the to-be-processed data according to the file descriptor of the to-be-processed data comprises: the solution layer sending a node configuration instruction to the node management module; the node management module configuring nodes for the to-be-processed data based on the node configuration instruction, and sending a first connection instruction to the pipeline module; the pipeline module connecting the nodes for the to-be-processed data into a pipeline based on the first connection instruction.

8. An electronic device, comprising: comprising: a memory and a processor; one or more computer programs stored in the memory, the one or more computer programs including instructions that, when executed by the processor, cause the electronic device to perform the method of any one of claims 1-7.

9. A computer storage medium, characterized in that computer instructions that, when executed by an electronic device, cause the electronic device to perform the method of any one of claims 1-7.

Images