Processing method of photographic data and related device

Abstract

The application provides a photographing data processing method and related equipment, which can be applied to an electronic device provided with a camera. According to the method, a target algorithm can be determined based on system information and algorithm information, so that the resource requirement of the target algorithm matches the system resource state, reduces the situation of load imbalance, reduces the freezing of a foreground application, and improves the user experience.

Description

Technical Field

[0001] This application belongs to the field of electronic devices, and in particular relates to a method for processing captured data and related equipment. Background Technology

[0002] With the increasing intelligence of electronic devices, users can install various applications to meet their daily life and work needs. For example, users can install camera applications to meet certain shooting needs (such as taking photos or videos).

[0003] However, as camera applications continue to iterate and develop, the algorithms involved in camera applications are becoming increasingly complex. As a result, when running camera applications, foreground applications often experience lag, which affects the user experience. Summary of the Invention

[0004] This application provides a method and related equipment for processing shooting data, which allows for more flexible selection of algorithms for processing shooting data, thereby achieving load balancing, improving the smoothness of the foreground application during camera application operation, and enhancing the user experience.

[0005] Firstly, a method for processing captured data is provided, applied to an electronic device or a module within an electronic device. The method includes: in response to a shooting operation, acquiring first system information and first algorithm information, and capturing image data or video data. Further, a target algorithm is determined from an algorithm set based on the first system information and the first algorithm information, and the image data or video data is processed according to the target algorithm. The first system information indicates the current first system resource status of the electronic device, the first algorithm information indicates the resource requirement corresponding to each algorithm in the algorithm set, and the resource requirement of the target algorithm matches the first system resource status.

[0006] Based on the above scheme, before determining the target algorithm, first system information can be obtained to determine the first system resource status. This first system resource status indicates the consumption of system resources or the system status. For example, the first system resource status includes one or more of the following: processor load, remaining memory, heat generation, power consumption, battery level, and sleep state. On the other hand, algorithm information is obtained to determine the resource requirements, or resource utilization, of one or more algorithms within the algorithm set. For example, this resource requirement includes one or more of the following: processor (e.g., central processing unit (CPU)) utilization, memory utilization, and heterogeneous hardware (e.g., graphics processing unit (GPU)) utilization. After obtaining the first system information and the first algorithm information, the target algorithm is determined from the algorithm set based on these information. The algorithms in this set are post-processing algorithms used to process image or video data, and the resource requirements of the target algorithm match the system resource status. In other words, the above scheme selects a target algorithm that matches the system resource status based on the system status and the algorithm's resource requirements, and then uses this target algorithm to process the captured image and video data. Matching the resource requirements of the target algorithm with the system resource status means that using the target algorithm will not cause system resource oversaturation, or in other words, the remaining system resources can cover the amount of resources required by the target algorithm. For example, the resource requirements of the target algorithm do not exceed the remaining system resources, and the target algorithm will not cause other resources of the electronic device (such as heat generation, power consumption, etc.) to exceed or fall below preset thresholds. In another implementation, matching the resource requirements of the target algorithm with the system resource status means that, provided that sufficient resources are reserved for the foreground application, the target algorithm will not cause system resource oversaturation.

[0007] Therefore, the decision logic of the algorithm in the method provided in this application embodiment has been optimized to determine the target algorithm that is suitable for the current scene and the current system resource status. This reduces the situation where the system resources are tight and the processing effect is poor when the target algorithm is used to process image data or video data, or other algorithms (such as preview algorithms or scene perception algorithms) cannot schedule resources or cannot schedule resources in a timely manner, thereby achieving system load balancing.

[0008] Optionally, when the resource requirements of multiple algorithms in the algorithm set match the resource status of the first system, a target algorithm is determined from among these multiple algorithms based on preset rules. In other words, when the remaining system resources can support multiple algorithms, the target algorithm can be determined according to preset rules, such as selecting the target algorithm based on its execution effect, to improve the processing effect of the final image or video data.

[0009] Optionally, the method further includes: determining scheduling priority information based on first system information and first algorithm information; invoking multiple threads of the target algorithm based on the scheduling priority information; invoking multiple threads of the target algorithm based on the scheduling priority information; and processing image data or video data using the target algorithm, wherein the scheduling priority information is used to indicate the priority of one or more threads corresponding to the target algorithm.

[0010] Based on the above scheme, the scheduling priority of the thread corresponding to the target algorithm can be determined according to the first system information and the first algorithm information. In other words, the resource scheduling order can be determined according to the system resource status and the resource requirements of the target algorithm to improve the execution efficiency of the algorithm.

[0011] Optionally, the method further includes: placing the processing task corresponding to the target algorithm into a task queue; when the processing task of the target algorithm is executed, selecting multiple threads corresponding to the target algorithm from a pre-created thread pool.

[0012] In the above scheme, a thread pool and task queue mechanism are introduced to execute the specific post-processing process, thereby improving efficiency and reducing resource consumption. For example, a limited number of threads can be created and stored in the thread pool according to actual needs. Furthermore, after determining the target algorithm, each algorithm within the target algorithm is abstracted into a task and stored in the task queue. Each task carries the scheduling priority information of the corresponding target algorithm, and optionally, each task also carries the control parameters of the corresponding target algorithm. During execution, tasks are retrieved from the task queue, and idle threads are selected from the thread pool according to preset rules. Since threads in the thread pool can be pre-created, whenever a new task arrives, the thread pool can immediately arrange a thread to handle it without waiting for thread creation, thus improving system response speed. Moreover, threads in the thread pool can be reused when needed, thereby reducing the resource consumption caused by thread creation and destruction. In addition, the thread pool can improve thread manageability. Through the thread pool, we can uniformly manage the creation, adjustment, and monitoring of threads, thereby better controlling thread usage and preventing performance problems caused by excessive thread growth. In summary, using a thread pool mechanism to process algorithms is an efficient, flexible, and controllable task processing method, especially suitable for scenarios that require a large number of parallel processing tasks.

[0013] Optionally, the method further includes: determining control parameters based on the first system information and the first algorithm information, wherein the control parameters include frequency boosting parameters and core binding parameters corresponding to the target algorithm; and configuring the control parameters as attribute information of multiple threads corresponding to the target algorithm.

[0014] In the above scheme, control parameters can also be determined based on the first system information and the first algorithm information. These control parameters include, for example, frequency-increasing parameters and core-binding parameters corresponding to the target algorithm. The frequency-increasing parameters are used to adjust the CPU frequency when using the target algorithm for post-processing, and the core-binding parameters are used to perform core-binding processing when using the target algorithm for post-processing. It is understood that these control parameters are used to accelerate the post-processing process, and therefore have a significant impact on system resource usage. Therefore, determining the control parameters based on the first system information (i.e., system resource status) and the first algorithm information (i.e., the resource requirements of the target algorithm) can prevent resource oversaturation during the acceleration of the target algorithm, reduce the occurrence of untimely or unavailable resource scheduling in the foreground application, and improve the smoothness of the foreground application.

[0015] Optionally, this first system information may also be used to indicate the type of foreground application.

[0016] In the above scheme, the electronic device can also obtain the type of the foreground application (e.g., a game application, a live streaming application, or a communication application) to determine the target algorithm. Since different types of applications have vastly different resource requirements, considering the type of the foreground application allows for sufficient resource allocation when determining the target algorithm. For example, a game application requires more resource allocation, while a desktop application requires less. This reduces foreground application lag and improves the user experience.

[0017] Optionally, before determining the target algorithm in the algorithm set based on the first system information and the first algorithm information, the method further includes: acquiring scene perception information and determining the algorithm set based on the scene perception information, wherein the scene perception information includes one or more of the following: shooting mode, frame rate information, resolution information, and lighting conditions.

[0018] In the above scheme, the algorithm set can be determined based on scene perception information. For example, if the scene perception information indicates that the current subject is a person and the scene is nighttime, then the algorithm set may include post-processing algorithms such as face recognition algorithms, beautification algorithms, night mode algorithms, and noise reduction algorithms. In other words, the algorithms in this set are post-processing algorithms that may be applicable to the current scene. Therefore, the electronic device does not need to obtain algorithm information corresponding to all algorithms, but only needs to obtain algorithm information that matches the current scene, thereby reducing resource consumption.

[0019] Optionally, before determining the target algorithm in the algorithm set based on the first system information and the first algorithm information, the method further includes: verifying that the first system information and the first algorithm information have passed the verification.

[0020] In the above scheme, after acquiring system information and algorithm information, the electronic device can verify the first system information and the first algorithm information. For example, it can re-acquire the system information and algorithm information or obtain it from other means to verify the previously acquired first system information and first algorithm information, in order to determine whether the previously acquired first system information and first algorithm information are accurate. This method can improve the accuracy and stability of system information and algorithm information. For example, video recording scenarios (high frame rate video recording scenarios) have higher resource requirements than photo shooting scenarios, so system resources will be more strained. Improving the accuracy of the first system information and first algorithm information can further improve the matching degree between the target algorithm and the system resource state, or in other words, further ensure load balancing, thereby improving the smoothness of the final video and enhancing the user experience.

[0021] Optionally, processing the image data or video data according to the target algorithm includes: in response to switching the camera application to a background application, acquiring second system information and second algorithm information, wherein the second system information is used to indicate the second system resource status of the electronic device after satisfying the resource scheduling requirements of the foreground application, and the second algorithm information is used to indicate the resource requirement of the algorithm to be processed in the target algorithm; determining whether the resource requirement of the algorithm to be processed matches the second system resource status; and if the resource requirement of the algorithm to be processed matches the second system resource status, processing the image data or video data using the algorithm to be processed.

[0022] In the above solution, when the camera application switches to the background, it can determine the processing algorithm that meets the current operating conditions of the mobile phone system, while satisfying the resource scheduling needs of the foreground application. On the one hand, it can ensure the smoothness of the foreground application, and on the other hand, it can continue to process the shooting events that need to be processed as much as possible.

[0023] Secondly, a method for processing captured data is provided, applied to an electronic device. The method includes: in response to switching a camera application to a background application, acquiring second system information and second algorithm information, wherein the second system information indicates the second system resource status of the electronic device after satisfying the resource scheduling requirements of the foreground application, and the second algorithm information indicates the resource requirement of the algorithm to be processed in the target algorithm, the target algorithm being used to process captured image or video data; determining whether the resource requirement of the algorithm to be processed matches the second system resource status; and, if the resource requirement of the algorithm to be processed matches the second system resource status, using the algorithm to process the image or video data.

[0024] Through the above process, while meeting the resource scheduling needs of the foreground application, it is possible to decide which shooting events meet the current operating conditions of the mobile phone system for processing. On the one hand, this ensures the smoothness of the foreground application, and on the other hand, it allows for the continued processing of shooting events that are yet to be processed.

[0025] Optionally, before acquiring the second system information and the second algorithm information, the method further includes: in response to the shooting operation, acquiring first system information and first algorithm information, wherein the first system information is used to indicate the current first system resource status of the electronic device, and the first algorithm information is used to indicate the resource requirement corresponding to each algorithm in the algorithm set; determining a target algorithm in the algorithm set based on the first system information and the first algorithm information, wherein the resource requirement of the target algorithm matches the first system resource status; capturing image data or video data, and processing the image data or video data according to a portion of the algorithms in the target algorithm.

[0026] Optionally, the first system resource status includes one or more of the following: processor load, remaining memory, heat generation, power consumption, battery level, hibernation state, and foreground application scenario; the resource demand includes one or more of the following: processor utilization and memory usage.

[0027] Optionally, before determining the target algorithm in the algorithm set based on system information and algorithm information, the method further includes: acquiring scene perception information and determining a first algorithm set based on the scene perception information, wherein the scene perception information includes one or more of the following: shooting mode, frame rate information, resolution information, and lighting conditions.

[0028] Optionally, the method further includes: determining that the verification of the first system information and the first algorithm information has passed.

[0029] Optionally, the method further includes: determining scheduling priority information based on first system information and first algorithm information, wherein the scheduling priority information is used to indicate the priority of multiple threads corresponding to the target algorithm; and processing the captured image data or video data according to the target algorithm, including: calling multiple threads of the target algorithm in priority order and using the target algorithm to process the image data or video data.

[0030] Optionally, the method further includes: placing the processing task corresponding to the target algorithm into a task queue; when the processing task of the target algorithm is executed, selecting multiple threads corresponding to the target algorithm from a pre-created thread pool.

[0031] Optionally, the method further includes: determining control parameters based on the first system information and the first algorithm information, wherein the control parameters include frequency boosting parameters and core binding parameters corresponding to the target algorithm; and configuring the control parameters as attribute information of multiple threads corresponding to the target algorithm.

[0032] Thirdly, an electronic device is provided, specifically comprising: a decision module, configured to, in response to a shooting operation, acquire first system information and first algorithm information, wherein the first system information is used to indicate the current first system resource status of the electronic device, and the first algorithm information is used to indicate the resource requirement corresponding to each algorithm in an algorithm set; the decision module is further configured to determine a target algorithm in the algorithm set based on the first system information and the first algorithm information, wherein the resource requirement of the target algorithm matches the first system resource status; and a post-processing module, configured to capture image data or video data, and process the image data or video data according to the target algorithm.

[0033] Fourthly, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the computer program, the electronic device performs the steps of the data processing method described in any one of the first or second aspects above.

[0034] Fifthly, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, implements the steps of the data processing method described in any one of the first or second aspects above.

[0035] In a sixth aspect, a computer program product is provided, which, when run on an electronic device, causes the electronic device to execute the data processing method described in either the first or second aspect.

[0036] In a seventh aspect, a chip system is provided, the chip system including a processor coupled to a memory, the processor executing a computer program stored in the memory to implement the data processing method of any one of the first or second aspects described above.

[0037] The chip system can be a single chip or a chip module composed of multiple chips.

[0038] It is understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0039] Figure 1 A schematic diagram of a photographing scenario provided in an embodiment of this application is shown;

[0040] Figure 2 A schematic diagram illustrating a video recording scenario applicable to an embodiment of this application is shown;

[0041] Figure 3 A schematic flowchart of a data processing method 300 provided in an embodiment of this application is shown.

[0042] Figure 4 A schematic diagram illustrating another photographing scenario to which the embodiments of this application are applicable is shown;

[0043] Figure 5 A schematic flowchart of a data processing method 500 provided in an embodiment of this application is shown;

[0044] Figure 6 A schematic block diagram of a data processing method 600 provided in an embodiment of this application is shown;

[0045] Figure 7 A schematic block diagram of a data processing method 700 provided in an embodiment of this application is shown.

[0046] Figure 8 A software system architecture diagram of an electronic device provided in an embodiment of this application is shown;

[0047] Figure 9 A hardware architecture diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. To more clearly describe the embodiments of this application, some terms or technologies involved in the embodiments of this application will be briefly introduced first.

[0049] As camera applications evolve, their algorithms become increasingly complex, leading to greater resource demands. This can cause stuttering or lag during use, especially when recording video, where dropped frames may occur, significantly impacting the user experience.

[0050] Therefore, this application provides a method for processing captured data, where the captured data refers to image or video data captured by a camera application. This method allows for more flexible selection of algorithms for processing the captured data, thereby achieving load balancing, improving the smoothness of the foreground application while the camera application's post-processing algorithm is running, and enhancing the user experience. It is understood that the foreground application here can be the camera application itself, a system application (such as the system desktop or settings interface), or other third-party applications (such as live streaming applications, game applications, etc.). In other words, the solution provided by this application improves the smoothness of the foreground application when the camera application has completed capturing the image or video data but the post-processing process has not yet been completed.

[0051] This method can be applied to various electronic devices with cameras, including but not limited to mobile phones, tablets, and laptops. This application does not impose any special restrictions on the specific type of electronic device.

[0052] Specifically, in the data processing method provided in this application embodiment, a target algorithm for processing the data can be determined based on system information and algorithm information, so that the resource requirements of the determined target algorithm match the current system resource status of the electronic device. For example, the resource requirements of the target algorithm do not exceed the remaining resources of the electronic device, and will not cause other resources of the electronic device (such as heat generation, power, etc.) to exceed or fall below a preset threshold, thereby achieving load balancing, reducing the situation where the foreground application cannot schedule resources or the scheduled resources are insufficient, and improving the smoothness of the foreground application.

[0053] Optionally, the scheduling priority of the thread corresponding to the target algorithm can be determined based on system information and algorithm information; and a thread pool mechanism can be introduced to realize flexible resource scheduling of the system and improve the processing efficiency of the algorithm.

[0054] Alternatively, if one or more shooting events remain to be processed after the camera application is moved to the background, the remaining resources can be determined after the system resources are rescheduled to meet the resource requirements of the algorithm corresponding to the shooting events to be processed. If they are met, the shooting data of the shooting events to be processed can continue to be processed by the algorithm. Otherwise, processing can be paused to prioritize the smoothness of the foreground application and improve the user experience.

[0055] The data processing method described in this application can be applied to shooting scenarios and post-processing scenarios. Shooting scenarios may include scenarios where an electronic device takes pictures (photos or videos) in different shooting modes after opening the camera application, as well as shooting scenarios where other applications call the camera application to take pictures.

[0056] The scenarios in which an electronic device takes photos or videos in different shooting modes after opening the camera application can include both multi-lens shooting modes and single-lens shooting modes. Specifically, shooting modes may include, for example, face recognition, face unlock, portrait mode, photo (normal photo), video, short video, watermark, time-lapse photography, dynamic photos, high-resolution photo, video (normal video), 60fps video, slow motion shooting, portrait mode, large aperture, professional mode, and super macro mode.

[0057] Other scenarios where applications call the camera app to take pictures include video call scenarios, face recognition scenarios, face unlock scenarios, face payment scenarios, and scenarios where the photo / video function is called.

[0058] The following is a schematic diagram illustrating application scenarios of the data processing method provided in the embodiments of this application implemented by electronic devices, with examples.

[0059] Figure 1 A schematic diagram illustrating a photographing scenario applicable to an embodiment of this application is shown. Wherein, Figure 1 (a) in the figure represents a user interface 11 for a camera application to perform a photo-taking action. In this embodiment, the camera application refers to an application installed in an electronic device that has photo-taking and / or video-recording functions. It is understood that the camera application can complete the shooting operation by calling the camera, which can be a camera located within the electronic device (such as the front or rear camera of the electronic device), or an external camera of the electronic device (such as the camera of an external camera connected to the electronic device via a data cable). This application does not limit the scope of the application to this.

[0060] like Figure 1 As shown in (a), the user interface 11 may include a camera control 111 and a preview window 112.

[0061] The camera control 111 can be used to receive camera-taking operations triggered by the user. In various shooting scenarios (including shooting mode, portrait mode, and night scene mode), the aforementioned camera-taking operation is the operation performed by the camera control 111 to control the taking of a photo. For example, the user can tap the camera control 111 to trigger the camera application to take a photo. It is understood that the aforementioned camera-taking operation can also be other types of operations, such as voice or gesture commands input by the user to instruct the user to take a photo, or other shortcut operations (such as long-pressing the power button), etc., which are not limited in this application.

[0062] The preview window 112 can be used to display the image sequence captured by the camera in real time. The image displayed in the preview window 112 can be called the raw image, that is, the image that has not been processed by post-processing algorithms. Users can view the real-time captured image from the camera through the preview window 112, such as... Figure 1The image of the puppy shown in (a) is an example. It is understood that when the subject of the camera is dynamically changing (such as a moving puppy), the preview window 112 displays the dynamically changing image captured by the camera in real time. It is also understood that the image sequence displayed in the preview window 112 is an image obtained after processing by a preview algorithm. This preview algorithm is used to process the image sequence captured by the camera in real time so as to display the processed preview image in real time. This preview algorithm may include, for example, beautification algorithms (specifically, face recognition algorithms, face slimming algorithms, whitening algorithms, eye enlargement algorithms, etc.), night scene algorithms, noise reduction algorithms, etc., which are not limited in this application. In one implementation, the preview algorithm can be determined based on scene perception information. This scene perception information can be determined by a scene perception algorithm. This scene perception information is used to indicate the currently captured object and / or scene, such as the scene recognition result indicating that the currently captured object is a puppy.

[0063] Optionally, the user interface 11 may also include a switching control 113, a menu bar 114, a playback control 115, a zoom control 116, a settings bar 117, etc.

[0064] The switching control 113 can be used to switch the currently used camera. If the camera currently used for image acquisition is the front-facing camera, upon detecting a user operation on the switching control 113, the electronic device can activate the rear-facing camera to acquire images in response to the operation. Conversely, if the camera currently used for image acquisition is the rear-facing camera, upon detecting a user operation on the switching control 113, the electronic device can activate the front-facing camera to acquire images in response to the operation.

[0065] Menu bar 114 displays multiple shooting mode options, such as Night Scene, Video, Photo, and Portrait modes. Night Scene mode is suitable for taking photos in low-light conditions, such as at night. Video mode is for recording videos. Photo mode is suitable for taking photos in daylight. Portrait mode is suitable for taking close-up photos of people. It's understandable that different shooting modes use different algorithms. The electronic device can determine the shooting mode based on user input or automatically select the mode through scene recognition.

[0066] The playback control 115 can be used to view previously captured photos or videos. Generally, the playback control 115 can display a thumbnail of a previously captured photo or a thumbnail of the first frame of a previously captured video.

[0067] The zoom control 116 can be used to adjust the magnification of the preview image within the preview window 112. For example, when the zoom control displays "1×", it indicates that the preview image is not magnified, and when the zoom control displays "2×", it indicates that the preview image has been magnified by two times. Users can click the zoom control 116 to switch between different magnification levels.

[0068] The settings panel 117 may display multiple shooting parameter setting controls (shooting controls). Each shooting control is used to set a type of camera parameter, thereby changing the image captured by the camera. For example, the settings panel 117 may display shooting controls such as aperture 1171, flash 1172, intelligent mode control 1173, and filter 1174. Aperture 1171 can be used to adjust the camera aperture size, thereby changing the brightness of the image captured by the camera; flash 1172 can be used to turn the flash on or off, thereby changing the brightness of the image captured by the camera; intelligent mode control 1173 can be used to activate the artificial intelligence shooting mode, which can automatically select shooting parameters based on artificial intelligence; filter 1174 can be used to select a filter style, thereby adjusting the image color. The settings panel 117 may also include a settings control 1175. Settings control 1175 can provide more controls for adjusting camera shooting parameters or image optimization parameters, such as white balance controls, ISO controls, beauty controls, body beautification controls, etc., thereby providing users with richer shooting services.

[0069] exist Figure 1 In the user interface 11 shown in (a), in response to receiving a shooting operation, the electronic device captures an image of a puppy. At this point, the user may not have stopped shooting but instead pointed the camera at other objects, such as... Figure 1 As shown in (b) in the diagram. Figure 1In the user interface 12 shown in (b), the preview window 122 displays a human portrait, while the playback control 125 displays a thumbnail of the previously captured image of a puppy. It is understandable that the thumbnail displayed in the playback control 125 may only be a thumbnail of the captured raw image data, and the electronic device may still be processing the raw image data in the background using post-processing algorithms. That is, due to the processing delay of post-processing algorithms, in some scenarios (especially heavy shooting, such as burst shooting), the image post-processing process and the camera preview process may be simultaneous. It is understood that the post-processing algorithms here are algorithms used to process the image data captured by the electronic device through the camera, such as beautification algorithms, high dynamic range (HDR) algorithms, noise reduction algorithms, age recognition algorithms, fill light algorithms, filter algorithms, mosaic algorithms, contrast algorithms, saturation algorithms, sharpening algorithms, background blur algorithms, etc. Post-processing algorithms are typically complex and therefore resource-intensive. Furthermore, to achieve optimal image quality and performance, the processing flow of post-processing algorithms is accelerated through software acceleration (such as thread concurrency and memory reuse) and hardware acceleration (such as CPU frequency scaling, core binding, and heterogeneous hardware acceleration (GPU, neural network processing unit (NPU), digital signal processor (DSP)). This can significantly consume system hardware and software resources, reducing the efficiency of lower-priority threads. For example, during camera preview, scene awareness and preview algorithms may be continuously invoked. Because post-processing consumes substantial resources, scene awareness and preview algorithms may face issues such as insufficient or untimely resource allocation and delayed hardware calls, resulting in preview stuttering and negatively impacting the user experience.

[0070] Figure 2 A schematic diagram illustrating a video recording scenario applicable to an embodiment of this application is shown. Wherein, Figure 2 (a) in the image represents a user interface 21 for a camera application performing a recording action. For example... Figure 2 As shown in (a), the user interface 21 may include a recording control 211 and a preview window 212.

[0071] The recording control 211 can be used to receive recording operations triggered by the user. In a recording scenario, the recording operation refers to the operation of controlling the recording via the recording control 211. Specifically, the recording control 211 includes a stop control 2111 and a start control 2112. The stop control 2111 is used to control the end of recording, and the start control 2112 is used to control the start and pause of recording. When the user clicks the start control 2112, the camera application starts recording; when the user clicks the stop control 2111, the camera application stops recording. It is understood that the recording operation can also be other types of operations, such as voice or gesture commands input by the user to instruct recording, or other shortcut operations (such as long-pressing the power button), etc., which are not limited in this application.

[0072] The preview window 212 can be used to display video captured by the camera in real time. The video displayed in the preview window 212 can be called the raw video, that is, the video that has not been processed by post-processing algorithms. Users can view the live footage captured by the camera through the preview window 212, such as... Figure 2 The video shows the person running, as shown in (a).

[0073] User interface 21 also includes a timing control 213, which is used to indicate the current shooting duration, for example, Figure 2 If the timer control 213 in (a) displays a time of 10, it means that it is the 10th second of the video recording.

[0074] exist Figure 2 In the user interface 21 shown, in response to the received end recording operation, the electronic device ends recording and obtains the recording data. At this time, the obtained recording data will be processed by a post-processing algorithm.

[0075] In one exemplary scenario, the user might continue to point the camera at other subjects, such as... Figure 2 As shown in (b) in the diagram. Figure 2 In the user interface 22 shown in (b), a person jumping rope is displayed in the preview window 222. At this time, the electronic device may be calling the scene awareness algorithm and the preview algorithm to process the image captured by the camera before displaying it in the preview window 223. That is to say, the post-processing process of video data and the camera preview process may be carried out simultaneously. Since the post-processing process will consume a large amount of system hardware and software resources, the scene awareness algorithm and the preview algorithm may face problems such as insufficient or untimely resource scheduling and untimely hardware calls, which will cause the preview screen to freeze and affect the user experience.

[0076] In another exemplary scenario, a user might open and watch a video file that was just recorded, such as... Figure 2As shown in (c) in the figure. Figure 2 The user interface 23 shown in (c) includes a display window 231 for displaying the video file. This user interface 23 may also include a progress bar 232 and control controls 233. The progress bar 232 displays the playback progress of the video file (including the total duration of the video file and the current playback duration), and the control controls 233 control the playback and pause operations of the video file, as well as retrieve the previous or next video for playback. Because the video data post-processing workflow requires a large amount of resources, insufficient system resources may cause video file corruption. Therefore, the video content played in the display window 231 may experience stuttering and frame drops, resulting in a poor user experience.

[0077] From the above Figure 1 and Figure 2 The example shown, and the analysis of that example, demonstrate that in scenarios involving taking photos or videos, post-processing algorithms can consume significant system resources, potentially leading to stuttering in the preview or dropped frames in the generated video file. Therefore, embodiments of this application provide a method for processing captured data that can improve load balancing. The following, in conjunction with... Figure 3 The process 300 in the document provides an example of the specific implementation process.

[0078] A1. Open the camera app.

[0079] For example, in response to detecting an action to open the camera app, the electronic device opens the camera app. For instance, in response to detecting a user tapping the camera app icon, the electronic device opens the main interface of the camera app. Or, for example, in response to detecting a user's voice command to "open the camera," the electronic device opens the main interface of the camera app.

[0080] B1. Perform filming.

[0081] For example, in response to a shooting operation, the electronic device performs a shooting action through a camera application. This shooting action can be a photograph or a video, and this application is not limited to this. Figure 1 Taking the user interface 11 shown in (a) as an example, in response to detecting that the user clicks the camera control 111, the electronic device performs a camera operation; Figure 2 Taking the user interface 21 shown in (a) as an example, the electronic device performs a recording operation in response to the user clicking the start control 2112.

[0082] C1. The electronic device determines the target algorithm and scheduling priority information through a decision module. It is understood that this decision module can be a newly added module or an existing module that has been enhanced and reused; this application does not limit this. The following provides an illustrative description of C1 with specific steps.

[0083] C11, A recording event was detected.

[0084] For example, after an electronic device takes a picture through a camera application, the decision module can listen for the shooting event (photo capture event or video recording event) and trigger C12 to C17, which will be explained below.

[0085] C12. Obtain scene-aware information.

[0086] For example, the decision module acquires scene perception information, which is used to indicate the currently captured object and / or scene. The captured object may be, for example, a person, an animal, or a landscape, and the scene may include, for example, day or night, lighting conditions, shooting mode, etc.

[0087] C13. Obtain system information and algorithm information.

[0088] Understandably, after the decision-making module detects a shooting event, it obtains system information and algorithm information.

[0089] The system information includes the current operating information of the system (such as remaining memory, heat generation, power consumption, hibernation status, etc.), and optionally, it also includes foreground application scenario information, which is used to indicate the type of foreground application, that is, what type of application is running in the foreground (for example, it can be a game application, a live streaming application, or a communication application).

[0090] The algorithm information includes the algorithm's resource utilization (such as CPU utilization, memory utilization, and GPU / other hardware utilization). Here, the algorithm refers to the post-processing algorithm used to process the captured image or video data. This algorithm information can be pre-configured, such as information pre-configured in the configuration repository or static memory. The algorithm's resource utilization can be an estimated value, an empirical value, or a value calculated based on the algorithm.

[0091] Optionally, in one possible implementation, the decision module can obtain algorithm information based on scene perception information. That is, the decision module can determine which algorithms' system resource utilization rates to obtain based on scene perception information, and these algorithms are collectively referred to as an algorithm set. The algorithms in this algorithm set are algorithms that match the scene perception information. For example, if the scene perception information indicates that the current shooting object is a person and the scene is nighttime, then the decision module can obtain the system resource utilization rates of post-processing algorithms such as face recognition algorithms, beautification algorithms, nighttime algorithms, and noise reduction algorithms.

[0092] C14. Verify system information and algorithm information.

[0093] Optionally, after acquiring system and algorithm information, the electronic device can verify the information, for example, by re-acquiring or obtaining it from other sources, to validate the information acquired in C13 and determine its accuracy. This improves the accuracy and stability of the system and algorithm information. For instance, video recording (high frame rate recording) demands more resources than still photography, leading to greater system resource constraints. Improving the accuracy of system and algorithm information further enhances the matching between the target algorithm and system resource status, or in other words, further ensures load balancing, thereby improving the smoothness of the final video and enhancing the user experience.

[0094] C15. Determine the target algorithm.

[0095] For example, the decision module determines the target algorithm based on system information and algorithm information, and the resource requirements of the target algorithm are matched with the system resource status. The resource requirements of the target algorithm are determined by the algorithm information, such as the CPU utilization, memory usage, and GPU / other hardware utilization of the target algorithm; the system resource status is determined by system information, such as remaining memory, heat generation, power consumption, remaining battery power, and sleep state.

[0096] Matching the resource requirements of the target algorithm with the system resource status means that the resource requirements of the target algorithm do not exceed the remaining system resources, and the target algorithm will not cause other resources of the electronic device (such as heat generation, power consumption, etc.) to exceed or fall below a preset threshold. In one implementation, the target algorithm is determined from a set of algorithms based on scene-aware information. The target algorithm includes one or more algorithms.

[0097] In other words, the decision logic of the algorithm in the method provided in this application embodiment has been optimized to determine the target algorithm that is suitable for the current scene and the current system resource status. This reduces the situation where the system resources are tight and the processing effect is poor when the target algorithm is used to process image data or video data, or other algorithms (such as preview algorithm or scene perception algorithm) cannot schedule resources or cannot schedule resources in time, thereby achieving system load balancing.

[0098] C16. Determine scheduling priority information.

[0099] Optionally, the decision module determines scheduling priority information based on system information and algorithm information. This scheduling priority information indicates the priority of one or more threads corresponding to the target algorithm. For example, the thread priority mechanism can be divided into three priority levels: high priority, medium priority, and low priority. This scheduling priority information can indicate at least one priority level for each algorithm in the target algorithm, and this priority level can be used for subsequent processing of the threads corresponding to the target algorithm.

[0100] Optionally, the decision module can also determine control parameters based on system information and algorithm information. These control parameters may include, for example, frequency boosting parameters and core binding parameters corresponding to the target algorithm. The frequency boosting parameters are used to adjust the CPU frequency when using the target algorithm for post-processing, and the core binding parameters are used to perform core binding processing when using the target algorithm for post-processing.

[0101] In summary, based on the above process, after detecting a shooting event, the decision module acquires system information and algorithm information, and determines the target algorithm and scheduling priority information based on the system information and algorithm information. This target algorithm and scheduling priority information can be used to process the shooting data (i.e., image data or video data). The processing procedure is illustrated below.

[0102] D1. Process the captured data according to the target algorithm and scheduling priority information.

[0103] For example, the decision module outputs the target algorithm and scheduling priority information to the execution module, which processes the captured image or video data according to the target algorithm.

[0104] This application does not limit the specific post-processing procedure. As a concrete example, a thread pool and task queue mechanism can be introduced to execute the specific process. For example, a limited number of threads can be created and stored in the thread pool according to actual needs. Furthermore, after determining the target algorithm, each algorithm within the target algorithm is abstracted into a task and stored in the task queue. Each task carries the scheduling priority information of the corresponding target algorithm; optionally, each task also carries the control parameters of the corresponding target algorithm. During execution, a task is retrieved from the task queue, an idle thread is selected from the thread pool according to preset rules, and the scheduling priority information and control parameters are retrieved from the task package and set into the thread attributes, thus initiating execution. Figure 3 D11 to D13 in the middle.

[0105] Therefore, this method introduces thread pools and task queues into the post-processing of captured data, which can improve algorithm execution efficiency and reduce resource consumption. Specifically, threads in the thread pool can be pre-created, so whenever a new task arrives, the thread pool can immediately assign a thread to handle it without waiting for thread creation, thus improving system response speed. Furthermore, threads in the thread pool can be reused when needed, reducing the resource consumption caused by thread creation and destruction. In addition, thread pools improve thread manageability; through thread pools, we can uniformly manage thread creation, adjustment, and monitoring, thereby better controlling thread usage and preventing performance issues caused by excessive thread growth. In summary, using a thread pool mechanism to process algorithms is an efficient, flexible, and controllable task processing method, particularly suitable for scenarios requiring a large number of parallel processing tasks.

[0106] in, Figure 4 (a) is a user interface 41 for a camera application to perform a photo-taking action; for details, please refer to [reference needed]. Figure 1 The description corresponding to user interface 11 in (a) of the document.

[0107] exist Figure 4 In the user interface 41 shown in (a), in response to receiving a shooting operation, the electronic device performs a photo-taking operation. Then, in response to the user's input application switching operation, the electronic device moves the camera application to the background (i.e., switches the camera application from the foreground application to the background application) and launches the game application (i.e., sets the game application to the foreground application), as shown in (a). Figure 4 User interface 42 is shown in (b) of the diagram.

[0108] It should be noted that the foreground application described in this application embodiment refers to an application currently running in the foreground of the electronic device. For example, when a user clicks the icon of a game application on the system desktop of an electronic device, the electronic device loads and displays the application interface of the game application on the screen. Until the user closes the game application or switches it to the background, the game application is the foreground application of the electronic device. It is understood that the foreground application described in this application embodiment can also refer to an application running in the background but perceptible to the user in the foreground, such as an input method being invoked by the user, music or video software playing sound in the background, or social or news applications appearing in the foreground notification bar via pop-ups. It is also understood that the foreground application described in this application embodiment can include system applications, such as system desktop applications, AI assistants appearing in the foreground via floating windows, and the system's negative one screen.

[0109] The background applications described in this application refer to applications that run in the background. Specifically, they can be applications that are switched from the foreground to the background by the user and whose processes remain resident in the background. For example, when a user clicks the camera application icon on the system desktop of an electronic device, the electronic device loads and displays the application interface of the camera application on the screen. When the user returns to the desktop from the application interface of the camera application or switches to other applications (without releasing the process of the camera application), the camera application is a background application of the electronic device. It is understood that the background applications described in this application can also include applications that run continuously in the background but are not perceived by the user, such as wireless fidelity (WiFi) and Bluetooth.

[0110] At this point, although the camera app has switched to the background, if there are still unprocessed photo-taking events—that is, the post-processing of the captured image or video data is not yet complete—the electronic device will continue to execute this post-processing process in the background. Therefore, the post-processing of image or video data may overlap with the runtime of the foreground application. Because post-processing algorithms consume significant system resources, the foreground application (such as…) will continue to run in the background. Figure 4 The game application shown in (b) may experience issues such as insufficient or untimely resource allocation, which can lead to lag in the foreground application and negatively impact the user experience.

[0111] Therefore, this application provides a method for processing captured data that prioritizes the resource needs of foreground applications, thereby improving the smoothness of foreground applications. The following is in conjunction with... Figure 5 The process described in step 500 is an example of the specific implementation process.

[0112] a1. Open the camera app.

[0113] b1. Perform filming.

[0114] The above two steps are similar to steps A1 and B1 in process 300, and will not be repeated here for the sake of brevity.

[0115] c1. Switch the camera app to the background.

[0116] For example, in response to detecting an operation that switches the camera app to the background, the electronic device switches the camera app to the background app.

[0117] d1. The electronic device determines whether to continue processing the shooting event through the decision module. The following is an example explanation of d1 with specific steps.

[0118] d11, switch the camera app to a background app.

[0119] For example, after an electronic device switches the camera application to the background, the decision module can listen to the event and trigger d11 to d17, which will be explained below.

[0120] d12, obtain system information and algorithm information.

[0121] For example, the decision module acquires system information and algorithm information. The system information indicates the system resource status of the electronic device after the resource scheduling requirements of the foreground application have been met. The foreground application can be a system application or other third-party applications (such as game applications), and this application does not limit this. The system information includes, for example, system operation information such as remaining memory, heat generation, power consumption, and hibernation status after the resource scheduling requirements of the foreground application have been met. Optionally, it also includes foreground application scenario information, which indicates the type of application running in the foreground.

[0122] The algorithm information indicates the resource requirements of the algorithm to be processed, such as the algorithm's CPU utilization, memory utilization, and GPU / other hardware utilization. The algorithm to be processed refers to the algorithm corresponding to the shooting event to be processed, or in other words, the algorithm used to process the image or video data corresponding to the shooting event to be processed. It is understood that this application does not limit the process of determining the algorithm corresponding to the shooting event to be processed. In one possible implementation, the algorithm can be determined using the scheme shown in step C15 of process 300.

[0123] d13 determines whether the CPU resources meet the algorithm's CPU resource utilization requirements.

[0124] For example, based on system information and algorithm information, it is determined whether the remaining CPU resources of the system can meet the CPU resource utilization of the algorithm to be processed after satisfying the resource scheduling requirements of the foreground application.

[0125] Understandably, when there are multiple pending shooting events, it can be determined sequentially whether the processing algorithm corresponding to each pending shooting event meets the above conditions. If none of the processing algorithms corresponding to all shooting events meet the conditions described in d13, then step d17 is executed directly, that is, the pending shooting events are temporarily not processed. This process continues until the system information is refreshed, at which point it is re-executed. Figure 3 or Figure 5 The process is shown below.

[0126] d14 determines whether the memory usage meets the algorithm's requirements.

[0127] For example, when at least one of the pending shooting events meets the judgment condition shown in d13, it can be added to the CPU list. Further, it is determined whether the remaining memory of the system after meeting the resource scheduling requirements of the foreground application meets the memory usage of at least one pending algorithm in the CPU list.

[0128] If none of the algorithms corresponding to all shooting events meet the conditions described in d14, then step d17 is executed directly, meaning the shooting events to be processed are temporarily not processed. This process continues until the system information is refreshed, at which point it is re-executed. Figure 3 or Figure 5 The process is shown below.

[0129] d15 determines whether other hardware resources in the system meet the hardware resource utilization rate of the algorithm.

[0130] For example, when at least one of the pending shooting events satisfies the judgment conditions shown in d13 and d14, it can be added to the corresponding list in memory. Further, it is then determined whether the remaining amount of other hardware resources (such as the remaining amount of GPU resources, the remaining power, etc.) after the system meets the resource scheduling requirements of the foreground application satisfies the other hardware resource occupancy rate of at least one pending algorithm in the memory list.

[0131] If none of the algorithms corresponding to all shooting events meet the conditions described in d15, then step d17 is executed directly, meaning the shooting events to be processed are temporarily not processed. This process continues until the system information is refreshed, at which point it is re-executed. Figure 3 or Figure 5 The process is shown below.

[0132] Through the above process, while meeting the resource scheduling needs of the foreground application, it is possible to decide which shooting events meet the current operating conditions of the mobile phone system for processing. On the one hand, this ensures the smoothness of the foreground application, and on the other hand, it allows for the continued processing of shooting events that are yet to be processed.

[0133] d16 determines the scheduling priority information.

[0134] Optionally, if at least one pending shooting event corresponds to a pending processing algorithm that satisfies the judgment conditions shown in d13, d14, and d15 above, the scheduling priority information corresponding to that pending processing algorithm is determined. This scheduling priority information is used to indicate the priority of one or more threads corresponding to that pending processing algorithm. The specific process is as follows... Figure 3 Step C16 in process 300 is similar and will not be repeated here.

[0135] e1. Process the captured data according to the target algorithm and scheduling priority information. Specific implementation methods are as follows: Figure 3 Step e1 in process 300 is similar and will not be repeated here.

[0136] Figure 6 An exemplary flowchart of a data processing method 600 provided in an embodiment of this application is shown. This method 600 can be applied to an electronic device or a module within an electronic device. For convenience, the description here uses an electronic device executing method 600 as an example. It is understood that method 600 can be used with... Figure 3 The process in question corresponds to 300.

[0137] S610, in response to shooting operation, acquires first system information and first algorithm information.

[0138] For example, after detecting a shooting operation, in response to the shooting operation, the electronic device acquires first system information and first algorithm information. The shooting operation is used to trigger the electronic device to call the camera to perform shooting (including taking a picture or recording a video), in order to... Figure 1 Taking the user interface 11 shown in (a) as an example, in response to detecting a user clicking the camera control 111, the electronic device acquires first system information and first algorithm information. Figure 2 Taking the user interface 21 shown in (a) as an example, in response to the user clicking the start control 2112, the electronic device obtains the first system information and the first algorithm information.

[0139] The first system information is used to indicate the current first system resource status of the electronic device, that is, the first system resource status of the electronic device when the shooting operation is detected. The first system resource status includes, for example, one or more of the following: processor load, remaining memory, heat generation, power consumption, battery level, sleep state, and foreground application scenario.

[0140] Optionally, the first system information is also used to indicate the type of foreground application, that is, what type of application is running in the foreground (e.g., it could be a game application, a live streaming application, or a communication application).

[0141] The first algorithm information is used to indicate the resource requirements corresponding to each algorithm in the algorithm set. The resource requirements include, for example, one or more of the following: processor (such as CPU) utilization, memory usage, and utilization of other heterogeneous hardware (such as GPU).

[0142] Optionally, the algorithms within this algorithm set are adapted to the current shooting scene. For example, the electronic device can first acquire scene perception information and determine the algorithm set based on this information. This scene perception information includes one or more of the following: shooting mode, frame rate information, resolution information, and lighting conditions. The algorithms within the algorithm set match this scene perception information. For instance, if the scene perception information indicates that the current shooting mode is portrait mode and the lighting conditions are nighttime, then the algorithm set may include post-processing algorithms such as portrait algorithms and nighttime algorithms.

[0143] Optionally, after acquiring the first system information and the first algorithm information, the electronic device can verify the first system information and the first algorithm information. For example, it can re-acquire the first system information and the first algorithm information from other sources to verify the previously acquired information. Subsequent processes are only executed if the verification passes. This approach can improve the accuracy and stability of the first system information and the first algorithm information.

[0144] Optionally, the electronic device can also determine scheduling priority information based on the first system information and the first algorithm information. This scheduling priority information is used to indicate the priority of one or more threads corresponding to the target algorithm. For example, the thread priority mechanism can be divided into three priority levels: high priority, medium priority, and low priority. The scheduling priority information can indicate at least one priority level corresponding to each algorithm in the target algorithm, and this priority level can be used for subsequent processing of the threads corresponding to the target algorithm.

[0145] Optionally, the electronic device can also determine control parameters based on the first system information and the first algorithm information. These control parameters include frequency boosting parameters and core binding parameters corresponding to the target algorithm. The frequency boosting parameters are used to adjust the CPU frequency when using the target algorithm for post-processing, and the core binding parameters are used to perform core binding processing when using the target algorithm for post-processing.

[0146] It is understandable that when method 600 corresponds to process 300, step S610 can be derived from, for example: Figure 3 The decision-making module shown is executed.

[0147] S620, determine the target algorithm in the algorithm set based on the first system information and the first algorithm information.

[0148] For example, after acquiring the first system information and the first algorithm information, the electronic device determines a target algorithm in the algorithm set based on the first system information and the first algorithm information, wherein the resource requirements of the target algorithm are matched with the resource status of the first system.

[0149] It is understandable that the resource requirements of the target algorithm are determined by the first algorithm information, such as the CPU utilization rate, memory utilization rate, and GPU and other hardware utilization rate of the target algorithm; the resource status of the first system is determined by the first system information, such as remaining memory, heat generation, power consumption, remaining battery power, and hibernation status.

[0150] Matching the resource requirements of the target algorithm with the resource status of the first system means that the resource requirements of the target algorithm do not exceed the remaining system resources, and the target algorithm will not cause other resources of the electronic device (such as heat generation, power consumption, etc.) to exceed or fall below a preset threshold. In one implementation, the target algorithm is determined from a set of algorithms based on scene-aware information. The target algorithm includes one or more algorithms.

[0151] The S630 captures image or video data and processes the image or video data according to the target algorithm.

[0152] For example, in response to a shooting operation, an electronic device acquires image data or video data by shooting. On the other hand, after determining a target algorithm, the electronic device processes the acquired image data or video data using that target algorithm.

[0153] This application does not limit the specific post-processing procedure. As a concrete example, a thread pool and task queue mechanism can be introduced to execute the specific process. For example, a limited number of threads can be created and stored in the thread pool according to actual needs. Furthermore, after determining the target algorithm, each algorithm within the target algorithm is abstracted into a task and stored in the task queue. Optionally, each task carries the scheduling priority information of the corresponding target algorithm. Optionally, each task also carries the control parameters of the corresponding target algorithm. During execution, tasks are retrieved from the task queue, idle threads are selected from the thread pool according to preset rules, and the scheduling priority information and control parameters are retrieved from the task package and set into the thread attributes.

[0154] Optionally, if the camera application is moved to the background, the electronic device can prioritize allocating resources for the foreground application and determine whether to continue executing the algorithm to be processed based on the remaining resources and the algorithm to be processed. For example, in response to switching the camera application to the background, the electronic device acquires second system information and second algorithm information. The second system information indicates the state of the second system resources after the resource scheduling requirements of the foreground application are met, and the second algorithm information indicates the resource requirements of the algorithm to be processed in the target algorithm. Then, it is determined whether the resource requirements of the algorithm to be processed match the state of the second system resources. For example, it is determined whether the system CPU resources can meet the CPU resource utilization of the algorithm to be processed; or whether the system memory can meet the memory utilization of the algorithm to be processed; or whether other heterogeneous hardware resources (such as GPU) can meet the utilization of these heterogeneous hardware resources by the algorithm to be processed. If the resource requirements of the algorithm to be processed match the state of the second system resources, the algorithm to be processed is used to process the image data or video data.

[0155] Figure 7 An exemplary flowchart of a data processing method 700 provided in an embodiment of this application is shown. This method 700 can be applied to an electronic device or a module within an electronic device. For convenience, the description here uses an electronic device executing method 700 as an example. It is understood that method 700 can be used with... Figure 3 The process in question corresponds to 300.

[0156] S710, in response to the operation of switching the camera application to a background application, obtains second system information and second algorithm information.

[0157] For example, after an electronic device opens the camera application and performs a shooting operation, if it detects an operation to switch the camera application to the background, in response to this operation, the electronic device obtains second system information and second algorithm information.

[0158] The second system information is used to indicate the second system resource status of the electronic device after the resource scheduling requirements of the foreground application are met, and the second algorithm information is used to indicate the resource requirements of the algorithm to be processed in the target algorithm. The target algorithm is used to process the captured image data or video data.

[0159] S720, determine whether the resource requirements of the algorithm to be processed match the resource status of the second system.

[0160] For example, it can determine whether the system's CPU resources are sufficient to meet the CPU resource utilization of the algorithm to be processed; or whether the system's memory is sufficient to meet the memory utilization of the algorithm to be processed; or whether other heterogeneous hardware resources (such as GPUs) are sufficient to meet the utilization of these heterogeneous hardware resources by the algorithm to be processed.

[0161] S730, when the resource requirements of the algorithm to be processed match the resource status of the second system, the algorithm to be processed is used to process the image data or video data.

[0162] Corresponding to the methods given in the above embodiments, this application also provides a schematic diagram of the structure of an electronic device to which the above methods can be applied. This electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture, etc. This application uses a layered Android system as an example to exemplify the software system of the electronic device.

[0163] Through the above process, while meeting the resource scheduling needs of the foreground application, it is possible to decide which shooting events meet the current operating conditions of the mobile phone system for processing. On the one hand, this ensures the smoothness of the foreground application, and on the other hand, it allows for the continued processing of shooting events that are yet to be processed.

[0164] It is understandable that method 700 can be implemented independently or in combination with method 600. When implemented in combination with method 600, the operations before the camera application is moved to the background can be referred to the description in method 600, which will not be repeated here.

[0165] See Figure 8 A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the software system is divided into five layers, from top to bottom: application layer, (application) framework layer, system layer, hardware abstraction layer, and driver layer. The hardware system includes the hardware layer.

[0166] The application layer may include a series of application packages. In this embodiment, the application package may include a camera, a gallery, etc.

[0167] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The framework layer includes some predefined functions. In this embodiment, the framework layer may include a camera access interface, which may include camera management and camera devices. The camera access interface is used to provide APIs and programming frameworks for camera applications.

[0168] The system layer includes the Android runtime, which comprises the core libraries and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system. The core libraries consist of two parts: the functions that Java code needs to call, and the core Android libraries. The application layer and application framework layer run in the virtual machine. The virtual machine executes the Java files in the application layer and application framework layer as binary files. The system layer contains system runtime libraries and system runtime information.

[0169] The hardware abstraction layer is an interface layer located between the application framework layer and the driver layer, providing a virtual hardware platform for the operating system. In this embodiment, the hardware abstraction layer may include a camera hardware abstraction layer and a camera algorithm library.

[0170] The camera hardware abstraction layer can provide virtual hardware for camera device 1, camera device 2, or more camera devices. The camera algorithm library may include runtime code and data that implement the shooting methods provided in the embodiments of this application.

[0171] The driver layer is the layer between hardware and software. It includes drivers for various hardware components, such as camera drivers, digital signal processor drivers, and image processor drivers.

[0172] The camera device driver is used to drive the camera sensor to acquire images and to drive the image signal processor to preprocess the images. The digital signal processor driver is used to drive the digital signal processor to process images. The image processor driver is used to drive the graphics processor to process images.

[0173] The following describes the shooting method in the embodiments of this application in detail, based on the above system structure:

[0174] In response to a user's action of opening the camera application, such as clicking the camera application icon, the camera application calls the camera access interface of the framework layer to launch the camera application. It then instructs the camera devices (camera device 1 and / or other camera devices) in the camera hardware abstraction layer to send a command to start the camera. The camera hardware abstraction layer triggers a decision module to determine the target algorithm. For example, the decision module obtains system information and algorithm information from the system runtime library, and then determines the target algorithm from the camera algorithm library based on this system information and algorithm information. The specific process can be found in C15 of process 300. Optionally, the decision module also determines scheduling priority information, the specific process of which can be found in C16 of process 300. It is understood that in practical applications, the decision module can also obtain system information and algorithm information through other images; this application does not limit this. For example, the decision module obtains the remaining memory from the system information through the system runtime library layer, because the system library contains core libraries used to support the operation of Android applications, such as graphics libraries, media libraries, and database libraries. On the other hand, the decision module can also call the API of the framework layer to obtain system information such as battery level and heat generation. Additionally, the decision module can also read algorithm information from the memory of the hardware layer. This application does not specify the specific process.

[0175] On the other hand, the camera hardware abstraction layer sends this instruction to the camera device driver in the kernel layer. This camera device driver can then activate the corresponding camera sensor and acquire image light signals through the sensor. One camera device in the camera hardware abstraction layer corresponds to one camera sensor in the hardware layer.

[0176] Then, the camera sensor can transmit the acquired image light signal to the image signal processor for preprocessing to obtain the image electrical signal (raw image), and transmit the raw image to the camera hardware abstraction layer through the camera device driver.

[0177] The camera hardware abstraction layer (HAL) sends the raw image to the post-processing module. Based on the target algorithm output by the decision module and scheduling priority information, the post-processing module retrieves the target algorithm from the camera algorithm library and processes the raw image. The post-processing module then sends the processed image, according to the target algorithm, back to the HAL. Finally, the HAL can display the image.

[0178] Corresponding to the methods given in the above embodiments, this application also provides a hardware architecture for a corresponding electronic device.

[0179] For example, Figure 9 A detailed architectural diagram of an electronic device 900 to which this application applies is shown.

[0180] like Figure 9As shown, the electronic device 900 may include a processor 910, one or more cameras 920 (these multiple displays may be represented by 1 to N, where N is a positive integer greater than 1), one or more displays 930 (these multiple displays may be represented by 1 to N), internal memory 940, sensor module 950, and audio module 960, wherein the audio module 960 includes at least a speaker 960A and a microphone 960B.

[0181] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 900. In other embodiments of this application, the electronic device 900 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0182] Processor 910 may include one or more processing units, such as an application processor (AP), a modem processor, a GPU, an image signal processor (ISP), a controller, a video codec, a DSP, a baseband processor, and / or an NPU. These different processing units may be independent devices or integrated into one or more processors. Processor 910 may also include memory for storing instructions and data.

[0183] Electronic devices implement display functions through a GPU, a display screen 930, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 930 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. The processor 910 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0184] Display screen 930 is used to display images, videos, etc. Display screen 930 includes a display panel. The display panel may be a liquid crystal display (LCD). In some embodiments, the electronic device may include one or N displays 930, where N is a positive integer greater than 1.

[0185] In this embodiment of the application, the electronic device displays the original image / video captured by the camera, the preview image / video, and the image / video obtained after post-processing by the algorithm, as well as... Figure 1 , Figure 2 , Figure 4 The capabilities of the user interface shown depend on the aforementioned GPU, display 930, and the display functions provided by the application processor.

[0186] Electronic devices can achieve shooting functions through an ISP, camera 920, video codec, GPU, display 930, and application processor. The ISP is used to process the data fed back by the camera 920.

[0187] Camera 920 is used to capture still images or videos. An object is projected onto a photosensitive element by an optical image generated through a lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device may include one or N cameras 920, where N is a positive integer greater than 1.

[0188] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when an electronic device is selecting a frequency, a DSP can perform a Fourier transform on the frequency energy.

[0189] Video codecs are used to compress or decompress digital video. Electronic devices can support one or more video codecs. This allows the electronic device to play or record video in various encoded formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0190] In this embodiment, the electronic device implements the image processing method provided in this embodiment, which first relies on the ISP (Image Signal Processor) and the image captured by the camera 920, and secondly on the video codec and the image computing and processing capabilities provided by the GPU. The electronic device can implement various post-processing algorithms through the computing power provided by the NPU (Network Processing Unit).

[0191] Internal memory 940 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM).

[0192] In this embodiment, the code implementing the data processing method described in this embodiment can be stored in non-volatile memory. When running the camera application or when the camera application is moved to the background, the electronic device can load the executable code stored in the non-volatile memory into random access memory.

[0193] Audio module 960 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. Speaker 960A is used to convert audio electrical signals into sound signals. Electronic devices can listen to music or make hands-free calls through speaker 960A. Microphone 960B is used to convert sound signals into electrical signals.

[0194] In this embodiment, while the electronic device is capturing images using the camera, it can simultaneously activate the microphone 960B to capture sound signals and convert the sound signals into electrical signals for storage. This allows the user to obtain video with sound.

[0195] In this embodiment, the electronic device can use a sensor module 950 (such as a touch sensor) to detect user actions such as clicking and swiping on the display screen 930, in order to achieve... Figure 1 , Figure 2 , Figure 4 The shooting process is shown below.

[0196] The term "user interface (UI)" used in this application refers to the medium through which an application or operating system interacts and exchanges information with the user. It converts the internal form of information into a form that the user can accept. The user interface of an application is source code written in a specific computer language such as Java or Extensible Markup Language (XML). This source code is parsed and rendered on the terminal device, ultimately presenting user-recognizable content, such as images, text, and buttons. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, images, and text. The attributes and content of controls in the interface are defined using tags or nodes, such as XML tags. <textview> 、 <imgview> 、 <videoview>Nodes define the controls contained in the interface. A node corresponds to a control or property in the interface, and after parsing and rendering, the node is presented as the content visible to the user. In addition, many applications, such as hybrid applications, often contain web pages within their interfaces. A web page, also known as a webpage, can be understood as a special control embedded in the application interface. Web pages are source code written in a specific computer language, such as Hypertext Markup Language (GTML), Cascading Style Sheets (CSS), JavaScript (JS), etc. Web page source code can be loaded and displayed as user-readable content by a browser or a web page display component with browser-like functionality. The specific content contained in a webpage is also defined through tags or nodes in the webpage source code; for example, GTML uses tags or nodes to define the content. 、 、 <video> 、 <canvas>To define the elements and attributes of a webpage.

[0197] The most common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0198] It is understood that if the units integrated in the above-described device embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above-described method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), RAM, electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0199] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0200] It should be noted that in the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0201] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0202] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

[0203] The terms "first," "second," "third," "fourth," and other various terminology (if present) used in the specification, claims, and accompanying drawings of this application are intended to distinguish similar objects and are not necessarily used to describe a particular order or quantity. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein.

[0204] In the various embodiments of this application, unless otherwise specified or logically conflicting, the terminology and / or descriptions between different embodiments are consistent and can be referenced mutually. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships. The specific operational methods in the method embodiments of this application can also be applied to the device embodiments or system embodiments.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A method for processing captured data, applied to electronic devices, characterized in that, The method includes: In response to a shooting operation, first system information and first algorithm information are acquired, wherein the first system information is used to indicate the current first system resource status of the electronic device, and the first algorithm information is used to indicate the resource requirement corresponding to each algorithm in the algorithm set; Based on the first system information and the first algorithm information, a target algorithm is determined from the algorithm set, and the resource requirements of the target algorithm are matched with the resource status of the first system. Image or video data is captured, and the image or video data is processed according to the target algorithm. The step of processing the image data or video data according to the target algorithm includes: in response to switching the camera application to a background application, acquiring second system information and second algorithm information, wherein the second system information is used to indicate the second system resource status of the electronic device after meeting the resource scheduling requirements of the foreground application, and the second algorithm information is used to indicate the resource requirement of the algorithm to be processed in the target algorithm; determining whether the resource requirement of the algorithm to be processed matches the second system resource status; and, if the resource requirement of the algorithm to be processed matches the second system resource status, processing the image data or video data using the algorithm to be processed.

2. The method according to claim 1, characterized in that, The method further includes: Scheduling priority information is determined based on the first system information and the first algorithm information, wherein the scheduling priority information is used to indicate the priority of one or more threads corresponding to the target algorithm; The process of processing the captured image or video data according to the target algorithm includes: Based on the scheduling priority information, multiple threads of the target algorithm are invoked to process the image data or video data.

3. The method according to claim 2, characterized in that, Before invoking the multiple threads of the target algorithm according to the scheduling priority information, the method further includes: Place the processing task corresponding to the target algorithm into the task queue; When the processing task of the target algorithm is executed, the multiple threads corresponding to the target algorithm are selected from the pre-created thread pool.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Control parameters are determined based on the first system information and the first algorithm information, and the control parameters include the frequency boosting parameters and core binding parameters corresponding to the target algorithm; The control parameters are configured as attribute information for multiple threads corresponding to the target algorithm.

5. The method according to any one of claims 1 to 3, characterized in that, The first system resource status includes one or more of the following: processor load, remaining memory, heat generation, power consumption, battery level, and sleep state; the resource demand includes one or more of the following: processor utilization, memory usage, and heterogeneous hardware utilization.

6. The method according to any one of claims 1 to 3, characterized in that, The first system information is also used to indicate the type of foreground application.

7. The method according to any one of claims 1 to 3, characterized in that, Before determining the target algorithm from the algorithm set based on the first system information and the first algorithm information, the method further includes: Acquire scene perception information and determine the algorithm set based on the scene perception information, wherein the scene perception information includes one or more of the following: shooting mode, frame rate information, resolution information, and lighting conditions.

8. The method according to any one of claims 1 to 3, characterized in that, Before determining the target algorithm from the algorithm set based on the first system information and the first algorithm information, the method further includes: The verification of the first system information and the first algorithm information has been completed.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory. The memory is used to store programs that support the electronic device in performing the methods provided by any one of claims 1 to 8, and to store data involved in implementing the methods described in any one of claims 1 to 8; The processor is configured to execute programs stored in the memory.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 8.

Images