Shooting data processing method and related equipment

Abstract

The invention provides a shooting data processing method and related equipment. The shooting data processing method can be applied to electronic equipment provided with a camera. According to the method, the target algorithm can be determined based on the system information and the algorithm information, so that the resource demand quantity of the target algorithm is matched with the system resource state, the condition of load imbalance is reduced, the blockage of a foreground application is reduced, and the user experience is improved.

Description

Technical Field

[0001] This application belongs to the field of electronic devices, and particularly relates to a method for processing shooting data and related devices. Background Art

[0002] With the intelligent development of electronic devices, users can install various applications in electronic devices to meet the needs of daily life and work. For example, users can install a camera application in an electronic device to meet some shooting requirements (such as taking pictures or recording videos).

[0003] However, with the continuous iteration and development of camera applications, the algorithms involved in camera applications are becoming increasingly complex. Therefore, when running a camera application, the foreground application often lags, affecting the user experience. Summary of the Invention

[0004] The embodiments of this application provide a method for processing shooting data and related devices, which can more flexibly select an algorithm for processing shooting data, so as to achieve load balancing, improve the fluency of the foreground application when the camera application is running, and improve the user experience.

[0005] In a first aspect, a method for processing shooting data is provided, which is applied to an electronic device or a module in an electronic device. The method includes: in response to a shooting operation, obtaining first system information and first algorithm information, and obtaining image data or video data by shooting. Further, determining a target algorithm in an algorithm set according to the first system information and the first algorithm information, and processing the image data or the video data according to the target algorithm. Wherein, the first system information is used to indicate the current first system resource state of the electronic device, the first algorithm information is used to indicate the resource demand corresponding to each algorithm in the algorithm set, and the resource demand of the target algorithm matches the first system resource state.

[0006] Based on the above solution, before determining the target algorithm, the first system information can be obtained to determine the first system resource state, which is used to indicate the consumption of system resources or the system state. For example, the first system resource state includes one or more of the following: processor load, remaining memory, heat generation, power consumption, battery level, sleep state, etc. On the other hand, algorithm information is obtained to determine the resource requirements, or resource occupancy rate, corresponding to one or more algorithms in the algorithm set. For example, the resource requirements include one or more of the following: occupancy rate of the processor (such as the central processing unit (CPU)), memory occupancy, occupancy rate of heterogeneous hardware (such as the graphics processing unit (GPU)). After obtaining the first system information and the first algorithm information, the target algorithm is determined from the algorithm set according to the first system information and the first algorithm information. The algorithms in the algorithm set are post-processing algorithms for processing image data or video data, and the resource requirements of the target algorithm match the system resource state. That is to say, the above solution selects a target algorithm that matches the system resource state based on the system state and the resource requirements of the algorithm, and uses the target algorithm to process the captured image data and video data. The matching of the resource requirements of the target algorithm and the system resource state means that using the target algorithm will not cause the system resources to be oversaturated. In other words, the remaining system resources can cover the resources to be used by the target algorithm. For example, the resource requirements of the target algorithm do not exceed the remaining system resources, and the target algorithm will not cause other resources of the electronic device (such as heat generation, battery level, etc.) to exceed or fall below the preset threshold. In another implementation, the matching of the resource requirements of the target algorithm and the system resource state means that the target algorithm will not cause the system resources to be oversaturated on the premise that sufficient resources are reserved for the foreground application.

[0007] Therefore, in the method provided by the embodiments of the present application, the decision logic of the algorithm is optimized to decide a target algorithm adapted to the current scenario and the current system resource state, reducing the situation that when the target algorithm is used to process image data or video data subsequently, the system resources are tense, resulting in poor processing effects, or other algorithms (such as preview algorithms or scene perception algorithms) cannot be scheduled for resources or cannot be scheduled for resources in a timely manner, and achieving the load balancing of the system.

[0008] Optionally, when the resource requirements of multiple algorithms in the algorithm set match the first system resource state, the target algorithm is determined from the multiple algorithms based on a preset rule. That is to say, when the remaining system resources can support multiple algorithms, the target algorithm can be determined according to the preset rule. For example, the target algorithm can be selected according to the execution effect of the algorithm to improve the processing effect of the final image data or video data.

[0009] Optionally, the method further includes: determining scheduling priority information according to the first system information and the first algorithm information, invoking multiple threads of the target algorithm according to the scheduling priority information, and processing the image data or video data by using the target algorithm, where the scheduling priority information is used to indicate the priorities of one or more threads corresponding to the target algorithm.

[0010] Based on the above solution, the scheduling priority of the threads corresponding to the target algorithm can be determined according to the first system information and the first algorithm information. That is, the resource scheduling order can be determined according to the system resource status and the resource requirements of the target algorithm, so as to improve the execution efficiency of the algorithm.

[0011] Optionally, the method further includes: putting the processing tasks corresponding to the target algorithm into a task queue; when the processing tasks of the target algorithm are executed, selecting multiple threads corresponding to the target algorithm from a pre-created thread pool.

[0012] In the above solution, a thread pool and a task queue mechanism are introduced to execute the specific post-processing process, thereby improving efficiency and reducing resource consumption. For example, a finite number of threads can be created according to actual needs and stored in the thread pool. In addition, after the target algorithm is determined, each algorithm in the target algorithm is abstracted into a task and stored in the task queue. Each task carries the scheduling priority information of the corresponding target algorithm. Optionally, each task also carries the control parameters of the corresponding target algorithm. During specific execution, a task is taken out from the task queue, and an idle thread is selected from the thread pool according to a preset rule. Since the threads in the thread pool can be created in advance, whenever a new task arrives, the thread pool can immediately arrange a thread to process it without waiting for the creation of the thread, thereby improving the system response speed. Moreover, the threads in the thread pool can be reused when needed, thus reducing the resource consumption caused by the creation and destruction of threads. In addition, the thread pool can also improve the manageability of threads. Through the thread pool, we can uniformly manage the creation, adjustment, and monitoring of threads, so as to better control the use of threads and prevent performance problems caused by excessive thread expansion. Generally speaking, processing algorithms through the thread pool mechanism is an efficient, flexible, and controllable task processing method, especially suitable for scenarios that require a large number of parallel processing tasks.

[0013] Optionally, the method further includes: determining control parameters according to the first system information and the first algorithm information, where the control parameters include frequency boosting parameters and core binding parameters corresponding to the target algorithm; configuring the control parameters as the attribute information of multiple threads corresponding to the target algorithm.

[0014] In the above solution, the control parameters can also be determined according to the first system information and the first algorithm information. The control parameters include, for example, the frequency boosting parameter and the core binding parameter corresponding to the target algorithm. The frequency boosting parameter is used to adjust the frequency of the CPU when using the target algorithm for the post-processing process, and the core binding parameter is used to perform core binding processing when using the target algorithm for the post-processing process. It can be understood that the control parameters are used to accelerate the post-processing process, so they have a great impact on the system resource occupancy. Therefore, determining the control parameters according to the first system information (i.e., the system resource status) and the first algorithm information (i.e., the resource demand of the target algorithm) can prevent resource oversaturation when accelerating the target algorithm, reduce the situation where the foreground application fails to schedule resources in a timely manner or fails to obtain resources, and improve the fluency of the foreground application.

[0015] Optionally, the first system information is further used to indicate the type of the foreground application.

[0016] In the above solution, the electronic device can also obtain the type of the foreground application (for example, it can be a game application, a live broadcast application, or a communication application) to determine the target algorithm. Since the resource demands of different types of applications vary greatly, considering the type of the foreground application can reserve sufficient resources for the foreground application when determining the target algorithm. For example, when the foreground application is a game application, more resources need to be reserved, and when the foreground application is a desktop application, only less resources need to be reserved. This reduces the situation of the foreground application freezing and improves the user experience.

[0017] Optionally, before determining the target algorithm from the algorithm set according to the first system information and the first algorithm information, the method further includes: obtaining scene perception information and determining the algorithm set according to the scene perception information, where the scene perception information includes one or more of the following: shooting mode, frame rate information, resolution information, and lighting conditions.

[0018] In the above solution, the algorithm set can be determined according to the scene perception information. For example, if the scene perception information indicates that the current shooting object is a person and the scene is at night, the algorithm set can include, for example, post-processing algorithms such as face recognition algorithm, beauty algorithm, night algorithm, and noise reduction algorithm. That is to say, the algorithms in the algorithm set are some post-processing algorithms that may be applicable to the current scene. Therefore, the electronic device does not need to obtain the algorithm information corresponding to all algorithms, but only needs to obtain the algorithm information matching the current scene, thereby reducing resource consumption.

[0019] Optionally, before determining the target algorithm from the algorithm set according to the first system information and the first algorithm information, the method further includes: determining that the verification of the first system information and the first algorithm information passes.

[0020] In the above solution, after the electronic device obtains the system information and algorithm information, it can verify the first system information and the first algorithm information. For example, it can obtain the system information and algorithm information again or from other channels to verify the first system information and the first algorithm information obtained previously, so as to determine whether the previously obtained first system information and first algorithm information are accurate. In this way, the accuracy and stability of the system information and algorithm information can be improved. For example, the video recording scenario (high frame rate video recording scenario) has higher requirements for resources than the photo-taking scenario, so the system resources will be more tense. Improving the accuracy of the first system information and the first algorithm information can further improve the matching degree between the target algorithm and the system resource status, or in other words, further ensure load balancing, thereby improving the smoothness of the final video and enhancing the user experience.

[0021] Optionally, processing the image data or video data according to the target algorithm includes: in response to an operation of switching the camera application to a background application, obtaining second system information and second algorithm information, where the second system information is used to indicate the second system resource status of the electronic device after meeting the resource scheduling requirements of the foreground application, and the second algorithm information is used to indicate the resource demand of the algorithm to be processed in the target algorithm; determining whether the resource demand of the algorithm to be processed matches the second system resource status; and in the case where the resource demand of the algorithm to be processed matches the second system resource status, processing the image data or video data using the algorithm to be processed.

[0022] In the above solution, when the camera application is switched to the background, on the premise of meeting the resource scheduling requirements of the foreground application, it can decide to process the algorithm to be processed that meets the current mobile phone system running conditions. On the one hand, it can ensure the smoothness of the foreground application, and on the other hand, it can try to continue processing the shooting event to be processed.

[0023] In a second aspect, a method for processing shooting data is provided, which is applied to an electronic device. The method includes: in response to an operation of switching the camera application to a background application, obtaining second system information and second algorithm information, where the second system information is used to indicate the second system resource status of the electronic device after meeting the resource scheduling requirements of the foreground application, and the second algorithm information is used to indicate the resource demand of the algorithm to be processed in the target algorithm, and the target algorithm is used to process the image data or video data obtained by shooting; determining whether the resource demand of the algorithm to be processed matches the second system resource status; and in the case where the resource demand of the algorithm to be processed matches the second system resource status, processing the image data or video data using the algorithm to be processed.

[0024] Through the above process, it is possible to make a decision on the shooting event that meets the current operating conditions of the mobile phone system on the premise of meeting the resource scheduling requirements of the foreground application, which can ensure the smoothness of the foreground application on the one hand, and on the other hand, can try to continue to process the shooting event to be processed.

[0025] Optionally, before obtaining the second system information and the second algorithm information, the method further includes: in response to a shooting operation, obtaining first system information and first algorithm information, where the first system information is used to indicate the current first system resource status of the electronic device, and the first algorithm information is used to indicate the resource requirements corresponding to each algorithm in the algorithm set; determining a target algorithm in the algorithm set according to the first system information and the first algorithm information, and the resource requirements of the target algorithm match the first system resource status; shooting to obtain image data or video data, and processing the image data or video data according to some algorithms in the target algorithm.

[0026] Optionally, the first system resource status includes one or more of the following: processor load, remaining memory, heat generation, power consumption, battery power, sleep state, foreground application scenario; the resource requirements include one or more of the following: processor occupancy, memory occupancy.

[0027] Optionally, before determining the target algorithm in the algorithm set according to the system information and the algorithm information, the method further includes: obtaining scene perception information, and determining a first algorithm set according to the scene perception information, where the scene perception information includes one or more of the following: shooting mode, frame rate information, resolution information, lighting condition.

[0028] Optionally, the method further includes: determining that the verification of the first system information and the first algorithm information passes.

[0029] Optionally, the method further includes: determining scheduling priority information according to the first system information and the first algorithm information, where the scheduling priority information is used to indicate the priorities of multiple threads corresponding to the target algorithm; processing the image data or video data obtained by shooting according to the target algorithm, including: calling multiple threads of the target algorithm in the order of priority, and using the target algorithm to process the image data or video data.

[0030] Optionally, the method further includes: putting the processing task corresponding to the target algorithm into a task queue; when executing the processing task of the target algorithm, selecting multiple threads corresponding to the target algorithm from a pre-created thread pool.

[0031] Optionally, the method further includes: determining control parameters according to the first system information and the first algorithm information, where the control parameters include frequency boosting parameters and core binding parameters corresponding to the target algorithm; configuring the control parameters as the attribute information of multiple threads corresponding to the target algorithm.

[0032] In a third aspect, an electronic device is provided, which specifically includes: a decision-making module, configured to obtain first system information and first algorithm information in response to a shooting operation, where the first system information is used to indicate the current first system resource status of the electronic device, and the first algorithm information is used to indicate the resource requirements corresponding to each algorithm in an algorithm set; the decision-making module is further configured to determine a target algorithm in the algorithm set according to the first system information and the first algorithm information, and the resource requirements of the target algorithm match the first system resource status; a post-processing module, configured to capture image data or video data, and process the image data or video data according to the target algorithm.

[0033] In a fourth aspect, an electronic device is provided, including a memory and a processor. A computer program that can run on the processor is stored on the memory. When the processor executes the computer program, the electronic device implements the steps of the method for processing shooting data described in any one of the first aspect or the second aspect above.

[0034] In a fifth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the method for processing shooting data described in any one of the first aspect or the second aspect above are implemented.

[0035] In a sixth aspect, a computer program product is provided. When the computer program product runs on an electronic device, the electronic device is caused to execute the method for processing shooting data described in any one of the first aspect or the second aspect above.

[0036] In a seventh aspect, a chip system is provided. The chip system includes a processor, and the processor is coupled to a memory. The processor executes a computer program stored in the memory to implement the method for processing shooting data described in any one of the first aspect or the second aspect above.

[0037] Wherein, the chip system may be a single chip or a chip module composed of multiple chips.

[0038] It can be understood that the beneficial effects of the second aspect to the seventh aspect above can refer to the relevant descriptions in the first aspect above, and will not be elaborated here. Description of the Drawings

[0039] Figure 1 A schematic diagram of a shooting scenario provided by an embodiment of the present application is shown;

[0040] Figure 2 A schematic diagram of a video recording scenario applicable to an embodiment of the present application is shown;

[0041] Figure 3 shows a schematic flowchart of a method 300 for processing captured data provided by an embodiment of the present application;

[0042] Figure 4 shows a schematic diagram of another photo-taking scenario applicable to an embodiment of the present application;

[0043] Figure 5 shows a schematic flowchart of a method 500 for processing captured data provided by an embodiment of the present application;

[0044] Figure 6 shows a schematic block diagram of a method 600 for processing captured data provided by an embodiment of the present application;

[0045] Figure 7 shows a schematic block diagram of a method 700 for processing captured data provided by an embodiment of the present application;

[0046] Figure 8 shows a software system architecture diagram of an electronic device provided by an embodiment of the present application;

[0047] Figure 9 shows a hardware architecture diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0048] Next, the technical solutions in the embodiments of the present application will be described with reference to the accompanying drawings in the embodiments of the present application. To describe the embodiments of the present application more clearly, some terms or technologies related to the embodiments of the present application will be briefly introduced first.

[0049] With the iterative development of camera applications, the algorithms involved in camera applications are becoming more and more complex, and the demand for resources is also increasing. Therefore, there may be a lag situation during the use of camera applications. Especially when shooting videos, there may even be a situation where the captured video lags and drops frames, greatly affecting the user experience.

[0050] Therefore, an embodiment of the present application provides a method for processing captured data, where the captured data refers to image data or video data obtained by shooting through a camera application. Through this method, the algorithm for processing captured data can be more flexibly selected, so as to achieve load balancing, improve the smoothness of the foreground application when the post-processing algorithm of the camera application is running, and improve the user experience. It can be understood that the foreground application here can be the camera application itself, or a system application (such as the system desktop or the settings interface, etc.), or other third-party applications (such as live broadcast applications, game applications, etc.). That is to say, the solution provided by the embodiment of the present application improves the smoothness of the foreground application when the post-processing process of the image data or video data captured by the camera application is not completed after the shooting is completed.

[0051] This method can be applied to various electronic devices with cameras, including but not limited to mobile phones, tablet computers, laptop computers, etc. The specific type of the electronic device is not particularly limited in the embodiments of the present application.

[0052] Specifically, in the method for processing captured data provided by the embodiments of the present application, a target algorithm for processing the captured data can be determined according to system information and algorithm information, so that the resource requirement of the determined target algorithm matches the current system resource state of the electronic device. For example, the resource requirement of the target algorithm does not exceed the remaining resources of the electronic device, and it will not cause other resources of the electronic device (such as heat generation, power consumption, etc.) to exceed or be lower than a preset threshold, thereby achieving load balancing and reducing the situation where foreground applications cannot be scheduled to resources or the scheduled resources are insufficient, and improving the fluency of foreground applications.

[0053] Optionally, the scheduling priority of the thread corresponding to the target algorithm can also be determined according to system information and algorithm information; and a thread pool mechanism is introduced to achieve flexible scheduling of resources in the system and improve the processing efficiency of the algorithm.

[0054] Further optionally, if there is still one or more pending capture events after the camera application has retreated to the background, it can be determined whether the remaining resources can meet the resource requirements of the algorithm corresponding to the pending capture events after the system resources are rescheduled; if so, the captured data of the pending capture events can continue to be processed through the algorithm, otherwise the processing can be paused to give priority to ensuring the fluency of foreground applications and improving the user experience.

[0055] The method for processing captured data described in the embodiments of the present application can be applied to capture scenarios and post-processing scenarios. The capture scenarios can include scenarios where the electronic device captures (takes pictures or records videos) in different capture modes after the camera application is launched, as well as capture scenarios where other applications call the camera application for capture.

[0056] The scenarios where the electronic device captures (takes pictures or records videos) in different capture modes after the camera application is launched can include both scenarios where the electronic device is in a multi-lens capture mode and scenarios where the electronic device is in a single-lens capture mode. Specifically, the capture modes can include, for example, face recognition, face unlocking, portrait, photo taking (ordinary photo taking), video recording, short video, watermark, time-lapse photography, live photos, high-pixel photo taking, video recording (ordinary video recording), 60fps video recording, slow-motion shooting, portrait mode, large aperture, professional, super macro, etc.

[0057] Scenarios where other applications call the camera application for shooting can include video call scenarios, face recognition scenarios, face unlocking scenarios, face payment scenarios, and photo / video shooting function call scenarios.

[0058] The following specifically introduces, with reference to examples, a schematic diagram of an application scenario for the electronic device to implement the method for processing shooting data provided in the embodiments of the present application.

[0059] Figure 1 A schematic diagram of a photo-taking scenario applicable to the embodiments of the present application is shown. Among them, Figure 1 in (a) is a user interface 11 for a camera application to perform a photo-taking action. The camera application in the embodiments of the present application refers to an application installed in an electronic device and having photo-taking and / or video-recording functions. It can be understood that the camera application can complete the shooting operation by calling the camera. Here, the camera can be a camera arranged in the electronic device (such as the front camera or rear camera of the electronic device), or an external camera of the electronic device (such as the camera of an external camera connected to the electronic device through a data cable). The present application does not make any limitations in this regard.

[0060] As Figure 1 shown in (a), the user interface 11 may include a photo-taking control 111 and a preview window 112.

[0061] Among them, the photo-taking control 111 can be used to receive a photo-taking operation triggered by the user. In a photo-taking scenario (including photo-taking mode, portrait mode, and night scene mode), the above photo-taking operation is an operation for controlling photo-taking acting on the photo-taking control 111. For example, the user clicking on the photo-taking control 111 with a hand can be used to trigger the camera application to take a photo. It can be understood that the above photo-taking operation can also be other types of operations. For example, a voice command or gesture command input by the user for indicating photo-taking, or other quick operations (such as long-pressing the power button), etc. The present application does not make any limitations in this regard.

[0062] The preview window 112 can be used to display the image sequence collected by the camera in real time. The image displayed in the preview window 112 can be called the original image, that is, an image that has not been processed by a post-processing algorithm. The user can view the real-time image collected by the camera through the preview window 112, such as Figure 1The puppy image shown in (a) of []. It can be understood that when the object captured by the camera is dynamically changing (such as a moving puppy), the preview window 112 displays the dynamically changing image captured by the camera in real time. It can also be understood that the image sequence displayed in the preview window 112 is the image obtained after being processed by the preview algorithm. The preview algorithm is used to perform real-time processing on the image sequence captured by the camera in real time so as to display the preview image after processing in real time. The preview algorithm may include, for example, a beauty algorithm (specifically, it may include a face recognition algorithm, a face slimming algorithm, a whitening algorithm, a big eye algorithm, etc.), a night scene algorithm, a noise reduction algorithm, etc., which are not limited in this application. In one implementation manner, the preview algorithm may be determined based on the scene perception information, and the scene perception information may be determined by the scene perception algorithm. The scene perception information is used to indicate the current captured object and / or scene. For example, the scene recognition result is used to indicate that the current captured object is a puppy.

[0063] Optionally, the user interface 11 may further include a conversion control 113, a menu bar 114, a review control 115, a zoom control 116, a setting bar 117, etc.

[0064] Among them, the conversion control 113 can be used to switch the currently used viewfinder camera. If the currently used camera for capturing images is the front camera, when a user operation acting on the conversion control 113 is detected, in response to the above operation, the electronic device can enable the rear camera to capture images. Conversely, if the currently used camera for capturing images is the rear camera, when a user operation acting on the conversion control 113 is detected, in response to the above operation, the electronic device can enable the front camera to capture images.

[0065] Multiple shooting mode options can be displayed in the menu bar 114, such as shooting modes like night scene, video recording, photo taking, portrait, etc. The night scene mode can be used to take photos in a scene with relatively dim light, such as taking photos at night. The video recording mode can be used to record videos. The photo taking mode can be used to take photos in a daylight scene. The portrait mode can be used to take close-up photos of people. It can be understood that the algorithms corresponding to different shooting modes are different. The electronic device can determine the shooting mode to be used based on the user's input, or can automatically select the shooting mode to be used through scene recognition.

[0066] The review control 115 can be used to view the previously taken photo or video. Generally, the review control 115 can display the thumbnail of the previously taken photo or the thumbnail of the first frame image of the previously taken video.

[0067] The zoom control 116 can be used to adjust the magnification of the preview image within the preview window 112. For example, when the zoom control shows "1×", it means the preview image is not magnified. When the zoom control shows "2×", it means the preview image is magnified by two times. The user can click on the zoom control 116 to switch between different magnifications.

[0068] Multiple shooting parameter setting controls (shooting controls) can be displayed in the setting bar 117. One shooting control is used to set a type of parameter of the camera, thereby changing the image captured by the camera. For example, the setting bar 117 can display shooting controls such as an aperture 1171, a flash 1172, an intelligent mode control 1173, and a filter 1174. The aperture 1171 can be used to adjust the aperture size of the camera, thereby changing the brightness of the image captured by the camera. The flash 1172 can be used to turn on or off the flash, thereby changing the brightness of the image captured by the camera. The intelligent mode control 1173 can be used to turn on the artificial intelligence shooting mode, which can automatically select shooting parameters based on artificial intelligence. The filter 1174 can be used to select a filter style and then adjust the image color. The setting bar 117 may also include a setting control 1175. The setting control 1175 can be used to provide more controls for adjusting the shooting parameters of the camera or image optimization parameters, such as a white balance control, an ISO control, a beauty control, a body shaping control, etc., thereby providing richer shooting services for the user.

[0069] In Figure 1 the user interface 11 shown in (a), in response to receiving a shooting operation, the electronic device captures an image of a puppy. At this time, the user may not have finished shooting but instead points the camera at other shooting objects, as shown in Figure 1 (b). In Figure 1In the user interface 12 shown in (b), the preview window 122 displays a portrait, while the thumbnail of the puppy image obtained in the previous shot is displayed within the review control 125. It can be understood that the thumbnail displayed within the review control 125 may only be a thumbnail of the captured raw image data, and the electronic device may still be processing the raw image data in the background through a post-processing algorithm. That is to say, due to the processing delay of the post-processing algorithm, in some scenarios (especially heavy shooting, such as continuous shooting scenarios), the post-processing process of the image and the camera preview process may be carried out synchronously. It can be understood that the post-processing algorithm here is an algorithm used to process the image data captured by the electronic device through the camera, for example, including beauty algorithms, high dynamic range (HDR) algorithms, noise reduction algorithms, age recognition algorithms, fill light algorithms, filter algorithms, mosaic algorithms, contrast algorithms, saturation algorithms, sharpening algorithms, background blurring algorithms, and so on. Since the post-processing algorithm is usually relatively complex, the demand for resources is relatively high. At the same time, in order to obtain the best photo-taking effect and performance experience, when operating the post-processing algorithm, the processing process of the algorithm will be accelerated, including software acceleration (such as thread concurrency, memory reuse, etc.) and hardware acceleration (such as CPU frequency modulation, core binding, heterogeneous hardware acceleration (GPU, neural-network processing unit (NPU), digital signal processor (DSP), etc.)). Therefore, this may significantly occupy the system's software and hardware resources, thereby reducing the operating efficiency of other low-priority threads. For example, during the camera preview process, the scene perception algorithm and the preview algorithm may be continuously called. However, due to the large amount of resources occupied by the post-processing process, the scene perception algorithm and the preview algorithm may face problems such as unavailable or untimely resource scheduling and untimely hardware calls, resulting in stuttering of the preview screen and affecting the user experience.

[0070] Figure 2 Fig. shows a schematic diagram of a video recording scenario applicable to the embodiments of the present application. Among them, Figure 2 In (a) is a user interface 21 of a camera application performing a video recording operation. As Figure 2 shown in (a), the user interface 21 may include a video recording control 211 and a preview window 212.

[0071] Among them, the video recording control 211 can be used to receive the video recording operation triggered by the user. In a video recording scenario, the above-mentioned video recording operation is an operation for controlling video recording on the video recording control 211. Specifically, for example, the video recording control 211 includes a stop control 2111 and a start control 2112. Among them, the stop control 2111 is used to control the end of video recording, and the start control 1221 is used to control the start of video recording and pause video recording. When it is detected that the user clicks the start control 2112, the camera application starts video recording. When it is detected that the user clicks the stop control 2111, the camera application ends video recording. It can be understood that the above-mentioned video recording operation can also be other types of operations. For example, a voice command or gesture command input by the user for indicating video recording, or other shortcut operations (such as long pressing the power button), etc. This application does not make any limitations in this regard.

[0072] The preview window 212 can be used to display the video captured by the camera in real time. The video displayed in the preview window 212 can be called the original video, that is, the video that has not been processed by the post-processing algorithm. The user can view the picture captured by the camera in real time through the preview window 212, such as Figure 2 the video of a person running as shown in (a) below.

[0073] The user interface 21 further includes a timing control 213, and this timing control 213 is used to indicate the current shooting duration. For example, Figure 2 if the time displayed by the timing control 213 in (a) below is 10, it means that this is the 10th second of video recording at this time.

[0074] In Figure 2 the user interface 21 shown, in response to the received operation of ending video recording, the electronic device ends video recording and obtains video data. At this time, the obtained video data will be processed by the post-processing algorithm.

[0075] In an exemplary scenario, the user may continue to aim the camera at other shooting objects, such as Figure 2 shown in (b) below. In Figure 2 the user interface 22 shown in (b) below, a person who is skipping rope is shown in the preview window 222. At this time, the electronic device may call the scene perception algorithm and the preview algorithm so as to display the picture captured by the camera after processing in the preview window 223. That is to say, the post-processing process of the video data and the camera preview process may be carried out synchronously. Since the post-processing process will greatly occupy the system's software and hardware resources, the scene perception algorithm and the preview algorithm may face problems such as insufficient or untimely resource scheduling and untimely hardware call, resulting in frame drops in the preview picture and affecting the user experience.

[0076] In another exemplary scenario, the user may open and view the video file just obtained through video recording, such as Figure 2as shown in (c) thereof. Figure 2 The user interface 23 shown in (c) thereof includes a display window 231 for displaying the picture of a video file. The user interface 23 may further include a progress bar 232 and control controls 233. The progress bar 232 is used to display the playing progress of the video file (including the total duration and the current playing duration of the video file), and the control controls 233 are used to control the playing and pausing operations of the video file, and are also used to retrieve the previous video or the next video for playing. Since the video data post-processing process needs to retrieve a large amount of resources, if the system resources are insufficient, it may cause the video file to be abnormal. Therefore, the video content played in the display window 231 may be stuck and frame dropped, resulting in a poor user experience.

[0077] From the above Figure 1 and Figure 2 shown example, and the analysis of this example, it can be seen that in the scenario of taking pictures or videos, the post-processing algorithm may occupy a large amount of system resources, which may cause the preview screen to be stuck or the generated video file to drop frames. In view of this, the embodiments of the present application provide a method for processing shooting data that can improve load balancing. The following combines Figure 3 the process 300 in it to make an exemplary description of the specific implementation process.

[0078] A1. Open the camera application.

[0079] Exemplarily, in response to detecting an operation to open the camera application, the electronic device opens the camera application. For example, in response to detecting an operation of the user clicking on the icon of the camera application, the electronic device opens the main interface of the camera application. For another example, in response to detecting a voice command of "open the camera" input by the user, the electronic device opens the main interface of the camera application.

[0080] B1. Perform shooting.

[0081] Exemplarily, in response to a shooting operation, the electronic device performs shooting through the camera application. Here, the shooting can be taking pictures or videos, which is not limited in the present application. Taking Figure 1 the user interface 11 shown in (a) thereof as an example, in response to detecting that the user clicks on the shooting control 111, the electronic device performs a shooting operation; taking Figure 2 the user interface 21 shown in (a) thereof as an example, in response to the user clicking on the start control 2112, the electronic device performs a video recording operation.

[0082] C1. The electronic device determines the target algorithm and scheduling priority information through a decision module. It can be understood that the decision module can be a newly added module or an existing module reused after being enhanced. This is not limited in the present application. The following makes an exemplary description of C1 in combination with specific steps.

[0083] C11. A shooting event is monitored.

[0084] Exemplarily, after the electronic device performs shooting through a camera application, the decision module can monitor the shooting event (a photo-taking event or a video-recording event), and then trigger C12 to C17, which will be described separately below.

[0085] C12. Scene perception information is obtained.

[0086] Exemplarily, the decision module obtains scene perception information, which is used to indicate the object and / or scene being currently shot. The shooting object can be, for example, a person, an animal, a landscape, etc. The scene includes, for example, day or night, lighting conditions, shooting mode, etc.

[0087] C13. System information and algorithm information are obtained.

[0088] It can be understood that after the decision module monitors the shooting event, it obtains system information and algorithm information.

[0089] Among them, the system information includes the current running information of the system (such as the remaining memory, heat generation, power consumption, sleep state, etc.). Optionally, it also includes foreground application scenario information, which is used to indicate the type of the foreground application, that is, what type of application is running in the foreground (for example, it can be a game application, a live broadcast application, or a communication application).

[0090] The algorithm information includes the occupancy rate of the algorithm on system resources (such as the occupancy rate of the algorithm on the CPU, the occupancy amount of the algorithm on the memory, the occupancy rate of the algorithm on hardware such as the GPU, etc.). Here, the algorithm refers to a post-processing algorithm used to process the captured image data or video data. The algorithm information can be pre-configured information, such as information pre-configured in a configuration bin or static memory. The occupancy rate of the algorithm on system resources can be an estimated value, or an empirical value, or a value calculated according to the algorithm.

[0091] Optionally, in a possible implementation manner, the decision module can obtain algorithm information based on the scene perception information, that is, the decision module can determine which algorithms' occupancy rates on system resources to obtain based on the scene perception information, and collectively refer to these algorithms as an algorithm set. The algorithms in this algorithm set are algorithms that match the scene perception information. For example, if the scene perception information indicates that the current shooting object is a person and the scene is at night, the decision module can obtain the occupancy rates of post-processing algorithms such as face recognition algorithm, beauty algorithm, night algorithm, noise reduction algorithm, etc. on system resources.

[0092] C14. The system information and algorithm information are verified.

[0093] Optionally, after obtaining the system information and algorithm information, the electronic device may verify the system information and algorithm information. For example, it may re-obtain or obtain the system information and algorithm information from other sources to verify the system information and algorithms obtained at C13, so as to determine whether the system information and algorithm information obtained at C13 are accurate. In this way, the accuracy and stability of the system information and algorithm information can be improved. For example, the video recording scenario (high frame rate video recording scenario) has higher resource requirements compared to the photo-taking scenario, so the system resources will be more strained. Improving the accuracy of the system information and algorithm information can further improve the matching degree between the target algorithm and the system resource state, or rather, further ensure load balancing, thereby improving the smoothness of the final video and enhancing the user experience.

[0094] C15. Determine the target algorithm.

[0095] Exemplarily, the decision module determines the target algorithm according to the system information and algorithm information, and the resource requirement of the target algorithm matches the system resource state. Among them, the resource requirement of the target algorithm is determined by the algorithm information, for example, including the CPU occupancy rate of the target algorithm, the memory occupancy of the target algorithm, the hardware occupancy rate of the GPU of the target algorithm, etc.; the system resource state is determined by the system information, for example, including the remaining memory, heat generation, power consumption, remaining battery power, sleep state, etc.

[0096] The fact that the resource requirement of the target algorithm matches the system resource state means that: the resource requirement of the target algorithm does not exceed the remaining system resources, and the target algorithm will not cause other resources of the electronic device (such as heat generation, power, etc.) to exceed or be lower than the preset threshold. In one implementation, the target algorithm is determined from the algorithm set determined based on the scene perception information. The target algorithm includes one or more algorithms.

[0097] That is to say, in the method provided by the embodiments of the present application, the decision logic of the algorithm is optimized to decide the target algorithm adapted to the current scene and the current system resource state, reducing the situation that when the target algorithm is used to process image data or video data subsequently, the system resources are strained, resulting in poor processing effects, or other algorithms (such as preview algorithms or scene perception algorithms) cannot be scheduled for resources or cannot be scheduled for resources in a timely manner, and achieving load balancing of the system.

[0098] C16. Determine the scheduling priority information.

[0099] Optionally, the decision-making module determines scheduling priority information based on system information and algorithm information, where the scheduling priority information is used to indicate the priority of one or more threads corresponding to the target algorithm. For example, the thread priority mechanism can divide into three priority levels, namely high priority, medium priority, and low priority. The scheduling priority information can indicate at least one priority level corresponding to each algorithm in the target algorithm, and this priority level can be used to process the threads corresponding to the target algorithm subsequently.

[0100] Optionally, the decision-making module can also determine control parameters based on system information and algorithm information. The control parameters include, for example, the frequency boosting parameter and core binding parameter corresponding to the target algorithm. The frequency boosting parameter is used to adjust the frequency of the CPU when using the target algorithm for post-processing, and the core binding parameter is used to perform core binding when using the target algorithm for post-processing.

[0101] In summary, based on the above process, after the decision-making module monitors a shooting event, it obtains system information and algorithm information, and determines the target algorithm and scheduling priority information based on the system information and algorithm information. The target algorithm and scheduling priority information can be used to process the shooting data (i.e., image data or video data). The following is an exemplary description of the processing process.

[0102] D1. Process the shooting data according to the target algorithm and scheduling priority information.

[0103] Exemplarily, the decision-making module outputs the target algorithm and scheduling priority information to the execution module, and the execution module is used to process the image data or video data obtained by shooting according to the target algorithm.

[0104] This application does not limit the specific post-processing process. As a specific example, a thread pool and task queue mechanism can be introduced to execute the specific process. For example, a finite number of threads can be created and stored in the thread pool according to actual needs. In addition, after determining the target algorithm, each algorithm in the target algorithm is abstracted into a task and stored in the task queue. Each task carries the scheduling priority information corresponding to the target algorithm. Optionally, each task also carries the control parameters corresponding to the target algorithm. During specific execution, a task is taken out from the task queue, an idle thread is selected from the thread pool according to a preset rule, and the scheduling priority information and control parameters are taken out from the task package and set into the thread attributes, that is, execute Figure 3 D11 - D13 in

[0105] Therefore, this method introduces the thread pool and task queue mechanisms into the post - processing process of the captured data, which can improve the algorithm execution efficiency and reduce resource consumption. Specifically, the threads in the thread pool can be pre - created. In this way, whenever a new task arrives, the thread pool can immediately arrange a thread to handle it without waiting for the creation of the thread, thus improving the system's response speed. Moreover, the threads in the thread pool can be reused when needed, thereby reducing the resource consumption caused by the creation and destruction of threads. In addition, the thread pool can also improve the manageability of threads. Through the thread pool, we can uniformly manage the creation, adjustment, and monitoring of threads, so as to better control the use of threads and prevent performance problems caused by excessive thread expansion. Generally speaking, processing algorithms through the thread pool mechanism is an efficient, flexible, and controllable task - processing method, especially suitable for scenarios that require a large number of parallel - processing tasks.

[0106] Among them, Figure 4 (a) of which is a user interface 41 for a camera application to perform a photographing action. For specific content, reference can be made to Figure 1 the description corresponding to the user interface 11 in (a) of

[0107] In Figure 4 In the user interface 41 shown in (a) of Figure 4 when receiving a photographing operation, the electronic device performs a photographing operation. Then, in response to the application switching operation input by the user, the electronic device moves the camera application to the background (i.e., switches the camera application from the foreground application to the background application) and starts the game application (i.e., sets the game application as the foreground application), as shown in the user interface 42 in (b) of

[0108] It should be noted that the foreground application described in the embodiments of this application refers to the application that is currently running in the foreground of the electronic device. For example, when the user clicks the icon of the game application on the system desktop of the electronic device, the electronic device will load and display the application interface of the game application on the display screen. Before the user closes the game application or switches the game application to the background, the game application is the foreground application of the electronic device. It can be understood that the foreground application described in the embodiments of this application can also refer to an application that is running in the background but can be perceived by the user in the foreground. For example, the input method being called by the user, the music or video software playing sound in the background, the social or news application that appears as a pop - up window in the foreground notification bar, etc. It can also be understood that the foreground application described in the embodiments of this application can include system applications. For example, the system desktop application, the artificial intelligence assistant that appears in the foreground through floating windows, the system negative - first screen, etc.

[0109] The background application described in the embodiments of the present application refers to an application that runs in the background. Specifically, it can refer to an application that is switched from the foreground to the background by the user and whose process remains resident in the background. For example, when the user clicks on the icon of the camera application on the system desktop of the electronic device, the electronic device will load and display the application interface of the camera application on the display screen. When the user returns from the application interface of the camera application to the desktop or switches to another application (without releasing the process of the camera application), the camera application becomes the background application of the electronic device. It can be understood that the background application described in the embodiments of the present application may also include applications that continuously run in the background but are not perceived by the user, such as wireless fidelity (WiFi), Bluetooth, etc.

[0110] At this time, although the camera application has been switched to a background application, if there are still unprocessed shooting events, that is, the post-processing process of the captured image data or video data has not been completed, the electronic device will continue to execute this post-processing process in the background. Therefore, the post-processing process of the image data or video data may overlap with the running time of the foreground application. Since the post-processing algorithm may consume excessive system resources, the foreground application (such as Figure 4 the game application shown in (b) in) may experience a situation where resources cannot be scheduled or the resource scheduling is not timely, resulting in the lag of the foreground application and affecting the user experience.

[0111] In view of this, the embodiments of the present application provide a method for processing shooting data that can preferentially ensure the resource requirements of the foreground application, which can improve the fluency of the foreground application. The following will make an exemplary description of the specific implementation process in conjunction with Figure 5 process 500 in.

[0112] a1. Open the camera application.

[0113] b1. Perform shooting.

[0114] The above two steps are similar to steps A1 and B1 in process 300. For the sake of brevity, they will not be elaborated here.

[0115] c1. Switch the camera application to the background.

[0116] Exemplarily, in response to detecting an operation to switch the camera application to the background, the electronic device switches the camera application to a background application.

[0117] d1. The electronic device determines whether to continue processing the shooting event through a decision module. The following will make an exemplary description of d1 in conjunction with specific steps.

[0118] d11. Monitor the switching of the camera application to a background application.

[0119] Exemplarily, after the electronic device switches the camera application to the background, the decision-making module can monitor this event, thereby triggering d11 to d17, which will be described separately below.

[0120] d12, obtain system information and algorithm information.

[0121] Exemplarily, the decision-making module obtains system information and algorithm information. Among them, the system information is used to indicate the system resource status after the electronic device meets the resource scheduling requirements of the foreground application. At this time, the foreground application can be a system application or other third-party applications (such as game applications), and the present application does not make any limitations in this regard. The system information includes, for example, system operation information such as the remaining memory, heat generation, power consumption, and sleep state after meeting the resource scheduling requirements of the foreground application. Optionally, it also includes foreground application scenario information, which is used to indicate the type of the foreground-running application.

[0122] The algorithm information is used to indicate the resource requirements of the algorithm to be processed, such as the CPU occupancy rate of the algorithm, the memory occupancy of the algorithm, and the hardware occupancy rates of the algorithm's GPU, etc. The algorithm to be processed refers to the algorithm corresponding to the shooting event to be processed, or in other words, the algorithm used to process the image data or video data corresponding to the shooting event to be processed. It can be understood that the present application does not limit the determination process of the algorithm corresponding to the shooting event to be processed. In one possible implementation manner, a scheme shown in step C15 of process 300 can be adopted to determine this algorithm.

[0123] d13, determine whether the CPU resources meet the CPU resource occupancy rate of the algorithm.

[0124] Exemplarily, based on the system information and algorithm information, determine whether the remaining CPU resources can meet the CPU resource occupancy rate of the algorithm to be processed after the system meets the resource scheduling requirements of the foreground application.

[0125] It can be understood that when there are multiple shooting events to be processed, it can be determined in sequence whether the algorithms to be processed corresponding to each shooting event to be processed meet the above conditions. If the algorithms to be processed corresponding to all shooting events do not meet the conditions described in d13, then directly execute step d17, that is, temporarily do not process the shooting events to be processed. Until the system information is refreshed, re-execute the Figure 3 or Figure 5 shown process.

[0126] d14, determine whether the memory meets the memory occupancy of the algorithm.

[0127] Exemplarily, when in the to-be-processed shooting event, the to-be-processed algorithms corresponding to at least one to-be-processed shooting event satisfy the judgment conditions shown in d13, they can be added to the list corresponding to the CPU (CPU list). Further, it is sequentially determined whether the remaining memory after the system meets the resource scheduling requirements of the foreground application satisfies the memory occupation of at least one to-be-processed algorithm in the CPU list.

[0128] If the to-be-processed algorithms corresponding to all shooting events do not satisfy the conditions described in d14, directly execute step d17, that is, temporarily do not process the to-be-processed shooting event. Until the system information is refreshed, re-execute Figure 3 or Figure 5 the process shown.

[0129] d15, determine whether other hardware resources of the system meet the hardware resource occupancy rate of the algorithm.

[0130] Exemplarily, when in the to-be-processed shooting event, the to-be-processed algorithms corresponding to at least one to-be-processed shooting event satisfy the judgment conditions shown in d13 and d14, they can be added to the list corresponding to the memory (memory list). Further, it is sequentially determined whether the remaining amount of other hardware resources (such as the remaining GPU resources, remaining battery power, etc.) after the system meets the resource scheduling requirements of the foreground application satisfies the occupancy rate of other hardware resources of at least one to-be-processed algorithm in the memory list.

[0131] If the to-be-processed algorithms corresponding to all shooting events do not satisfy the conditions described in d15, directly execute step d17, that is, temporarily do not process the to-be-processed shooting event. Until the system information is refreshed, re-execute Figure 3 or Figure 5 the process shown.

[0132] Through the above process, it is possible to decide which shooting events meet the current mobile phone system operation conditions for processing on the premise of meeting the resource scheduling requirements of the foreground application. On the one hand, it can ensure the fluency of the foreground application, and on the other hand, it can try to continue to process the to-be-processed shooting events.

[0133] d16, determine the scheduling priority information.

[0134] Optionally, in the case where there is at least one to-be-processed algorithm corresponding to a to-be-processed shooting event that satisfies the judgment conditions shown in d13, d14, and d15 above, determine the scheduling priority information corresponding to the to-be-processed algorithm, and the scheduling priority information is used to indicate the priority of one or more threads corresponding to the to-be-processed algorithm. The specific process is similar to the step C16 in the process 300 shown in Figure 3 and will not be elaborated here.

[0135] e1. Process the captured data according to the target algorithm and scheduling priority information. The specific implementation is similar to step e1 in process 300 in Figure 3 and will not be elaborated here.

[0136] Figure 6 FIG. shows an exemplary flowchart of a method 600 for processing captured data provided by an embodiment of the present application. The method 600 can be applied to an electronic device or a module in the electronic device. For convenience, the example of the electronic device executing the method 600 is used here for illustration. It can be understood that the method 600 can correspond to Figure 3 process 300 in

[0137] S610. In response to a shooting operation, obtain first system information and first algorithm information.

[0138] Exemplarily, after detecting the shooting operation, in response to the shooting operation, the electronic device obtains first system information and first algorithm information. Among them, the shooting operation is used to trigger the electronic device to call the camera for shooting (including taking pictures or recording videos). Taking the user interface 11 shown in Figure 1 (a) in Figure 2 as an example, in response to detecting the operation of the user clicking the photo-taking control 111, the electronic device obtains first system information and first algorithm information. Taking the user interface 21 shown in

[0139] (a) in

[0140] as an example, in response to the user clicking the start control 2112, the electronic device obtains first system information and first algorithm information.

[0141] Among them, the first system information is used to indicate the current first system resource state of the electronic device, that is, the first system resource state of the electronic device when the shooting operation is detected. The first system resource state includes, for example, one or more of the following information: processor load, remaining memory, heat generation, power consumption, battery power, sleep state, foreground application scenario.

[0142] Optionally, the algorithms in the algorithm set are adapted to the current shooting scene. For example, the electronic device can first obtain scene perception information and determine the algorithm set according to the scene perception information, where the scene perception information includes one or more of the following: shooting mode, frame rate information, resolution information, and lighting conditions, and the algorithms in the algorithm set match the scene perception information. For example, if the scene perception information indicates that the current shooting mode is the portrait mode and the lighting condition is night, the algorithm set may include post-processing algorithms such as portrait algorithms and night algorithms.

[0143] Optionally, after obtaining the first system information and the first algorithm information, the electronic device can verify the first system information and the first algorithm information. For example, the first system information and the first algorithm information are re-obtained or obtained from other channels to verify the previously obtained information. Only when the verification is passed, the subsequent process is executed. In this way, the accuracy and stability of the first system information and the first algorithm information can be improved.

[0144] Optionally, the electronic device can also determine scheduling priority information according to the first system information and the first algorithm information, and the scheduling priority information is used to indicate the priority of one or more threads corresponding to the target algorithm. For example, the thread priority mechanism can divide into three priority levels, namely high priority, medium priority, and low priority. The scheduling priority information can indicate at least one priority level corresponding to each algorithm in the target algorithm, and the priority level can be used to process the threads corresponding to the target algorithm subsequently.

[0145] Optionally, the electronic device can also determine control parameters according to the first system information and the first algorithm information, and the control parameters include the frequency boosting parameter and the core binding parameter corresponding to the target algorithm. Among them, the frequency boosting parameter is used to adjust the frequency of the CPU when using the target algorithm for post-processing, and the core binding parameter is used to perform core binding processing when using the target algorithm for post-processing.

[0146] It can be understood that when method 600 corresponds to process 300, step S610 can be executed by a decision module as shown in Figure 3 shown.

[0147] S620. Determine the target algorithm in the algorithm set according to the first system information and the first algorithm information.

[0148] Exemplarily, after obtaining the first system information and the first algorithm information, the electronic device determines the target algorithm in the algorithm set according to the first system information and the first algorithm information, where the resource requirement of the target algorithm matches the first system resource state.

[0149] It can be understood that the resource requirements of the target algorithm are determined by the first algorithm information, such as the CPU occupancy rate of the target algorithm, the memory occupancy of the target algorithm, the hardware occupancy rate of the GPU of the target algorithm, etc.; the first system resource state is determined by the first system information, such as the remaining memory, heat generation, power consumption, remaining battery power, sleep state, etc.

[0150] The matching of the resource requirements of the target algorithm and the first system resource state means that: the resource requirements of the target algorithm do not exceed the remaining system resources, and the target algorithm will not cause other resources of the electronic device (such as heat generation, power, etc.) to exceed or fall below the preset threshold. In one implementation, the target algorithm is determined from a set of algorithms determined based on scenario perception information. The target algorithm includes one or more algorithms.

[0151] S630, capture image data or video data, and process the image data or video data according to the target algorithm.

[0152] Exemplarily, in response to a capture operation, the electronic device captures image data or video data through shooting. On the other hand, after determining the target algorithm, the electronic device uses the target algorithm to process the captured image data or video data.

[0153] This application does not limit the specific post-processing process. As a specific example, a thread pool and a task queue mechanism can be introduced to execute the specific process. For example, a finite number of threads can be created and stored in the thread pool according to actual needs. In addition, after determining the target algorithm, each algorithm in the target algorithm is abstracted into a task and stored in the task queue. Optionally, each task carries the scheduling priority information of the corresponding target algorithm. Optionally, each task also carries the control parameters of the corresponding target algorithm. During specific execution, tasks are taken out from the task queue, idle threads are selected from the thread pool according to preset rules, and the scheduling priority information and control parameters are taken out from the task package and set into the thread attributes.

[0154] Optionally, if the camera application is switched to the background, the electronic device may preferentially schedule resources for the foreground application and determine whether to continue executing the pending algorithm based on the remaining resources and the pending algorithm. For example, in response to an operation of switching the camera application to a background application, the electronic device obtains second system information and second algorithm information, where the second system information is used to indicate the second system resource state of the electronic device after meeting the resource scheduling requirements of the foreground application, and the second algorithm information is used to indicate the resource demand of the pending algorithm in the target algorithm. Then, it is determined whether the resource demand of the pending algorithm matches the second system resource state. For example, it is determined whether the system CPU resources can meet the CPU resource occupancy rate required by the pending algorithm; for another example, it is determined whether the system memory can meet the memory occupancy amount required by the pending algorithm; for another example, it is determined whether other heterogeneous hardware resources (such as GPU) of the system can meet the occupancy rate of these heterogeneous hardware by the pending algorithm. When the resource demand of the pending algorithm matches the second system resource state, the pending algorithm is used to process the image data or video data.

[0155] Figure 7 FIG. 4 shows an exemplary flowchart of a method 700 for processing captured data provided by an embodiment of the present application. The method 700 may be applied to an electronic device or a module in the electronic device. For convenience, the method 700 is described herein by taking the electronic device as an example of executing the method 700. It can be understood that the method 700 may correspond to the Figure 3 process 300 in.

[0156] S710, in response to an operation of switching the camera application to a background application, obtain second system information and second algorithm information.

[0157] Exemplarily, after the electronic device turns on the camera application and performs a shooting operation, if it detects an operation of switching the camera application to a background application, in response to this operation, the electronic device obtains second system information and second algorithm information.

[0158] Wherein, the second system information is used to indicate the second system resource state of the electronic device after meeting the resource scheduling requirements of the foreground application, and the second algorithm information is used to indicate the resource demand of the pending algorithm in the target algorithm, and the target algorithm is used to process the captured image data or video data.

[0159] S720, determine whether the resource demand of the pending algorithm matches the second system resource state.

[0160] For example, determine whether the CPU resources of the system can meet the CPU resource occupancy rate required by the algorithm to be processed; for another example, determine whether the system memory can meet the memory occupancy of the algorithm to be processed; for yet another example, determine whether other heterogeneous hardware resources of the system (such as GPU) can meet the occupancy rate of these heterogeneous hardware by the algorithm to be processed.

[0161] S730, when the resource requirements of the algorithm to be processed match the second system resource status, use the algorithm to be processed to process the image data or video data.

[0162] Corresponding to the method given in the above embodiments, the present application also provides a schematic structural diagram of an electronic device that can apply the above method. The electronic device can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture, etc. In the embodiments of the present application, taking the Android system with a layered architecture as an example, the software system of the electronic device is exemplarily described.

[0163] Through the above process, it is possible to make a decision on a shooting event that meets the operating conditions of the current mobile phone system for processing on the premise of meeting the resource scheduling requirements of the foreground application. On the one hand, it can ensure the smoothness of the foreground application, and on the other hand, it can try to continue processing the shooting event to be processed.

[0164] It can be understood that method 700 can be implemented independently or in combination with method 600. When implemented in combination with method 600, the operations before the camera application retreats to the background can refer to the description in method 600 and will not be elaborated here.

[0165] See Figure 8 , the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the software system is divided into five layers, from top to bottom are the application layer, (application) framework layer, system layer, hardware abstraction layer, and driver layer, and the hardware system includes the hardware layer.

[0166] The application layer may include a series of application program packages. In the embodiments of the present application, the application program packages may include a camera, a gallery, etc.

[0167] The framework layer provides application programming interfaces (APIs) and programming frameworks for the application programs in the application layer. The framework layer includes some predefined functions. In the embodiments of the present application, the framework layer may include a camera access interface, where the camera access interface may include camera management and camera devices. The camera access interface is used to provide application programming interfaces and programming frameworks for the camera application.

[0168] The system layer includes the Android runtime, which consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system. The core libraries contain two parts: one part is the functional functions that need to be called by the Java language, and the other part is the core libraries of Android. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The system runtime library in the system layer runs system information.

[0169] The hardware abstraction layer is an interface layer located between the application framework layer and the driver layer, providing a virtual hardware platform for the operating system. In the embodiments of the present application, the hardware abstraction layer may include a camera hardware abstraction layer and a camera algorithm library.

[0170] Among them, the camera hardware abstraction layer can provide virtual hardware for camera device 1, camera device 2, or more camera devices. The camera algorithm library may include the running code and data for implementing the shooting method provided in the embodiments of the present application.

[0171] The driver layer is the layer between hardware and software. The driver layer includes drivers for various hardware. The driver layer may include a camera device driver, a digital signal processor driver, an image processor driver, etc.

[0172] Among them, the camera device driver is used to drive the sensor of the camera to collect images and drive the image signal processor to preprocess the images. The digital signal processor driver is used to drive the digital signal processor to process the images. The image processor driver is used to drive the graphics processor to process the images.

[0173] Next, in combination with the above system structure, the shooting method in the embodiments of the present application will be specifically described:

[0174] In response to the user's operation of opening the camera application, such as the operation of clicking on the camera application icon, the camera application calls the camera access interface of the framework layer to start the camera application, and then instructs these camera devices to send instructions to start the camera by calling the camera devices (Camera Device 1 and / or other camera devices) in the camera hardware abstraction layer. The camera hardware abstraction layer triggers the decision-making module to determine the target algorithm. For example, the decision-making module obtains system information and algorithm information from the system runtime library, and then determines the target algorithm from the camera algorithm library based on this system information and algorithm information. The specific process can refer to C15 in Flow 300. Optionally, the decision-making module also determines scheduling priority information, and the specific process can refer to C16 in Flow 300. It can be understood that in actual applications, the decision-making module can also obtain system information and algorithm information through other image acquisition systems, which is not limited in this application. For example, the decision-making module obtains the remaining memory in the system information through the system runtime library layer, because the system treasury contains core libraries for supporting the operation of Android applications, such as graphics libraries, media libraries, database libraries, etc. On the other hand, the decision-making module can also call the API of the framework layer to obtain system information such as battery power and heat generation. In addition, the decision-making module can also read algorithm information from the memory of the hardware layer. The specific process is not limited in this application.

[0175] On the other hand, the camera hardware abstraction layer sends this instruction to the camera device driver in the kernel layer. This camera device driver can start the corresponding camera sensor and collect image optical signals through the sensor. One camera device in the camera hardware abstraction layer corresponds to one camera sensor in the hardware layer.

[0176] Then, the camera sensor can transmit the collected image optical signals to the image signal processor for preprocessing to obtain image electrical signals (raw images), and transmit the above raw images to the camera hardware abstraction layer through the camera device driver.

[0177] The camera hardware abstraction layer can send the raw image to the post-processing module. The post-processing module obtains the target algorithm from the camera algorithm library based on the target algorithm and scheduling priority information output by the decision-making module, and processes the raw image. Subsequently, the post-processing module can send the image processed according to the target algorithm to the camera hardware abstraction layer. Then, the camera hardware abstraction layer can display it.

[0178] Corresponding to the methods given in the above method embodiments, the embodiments of the present application also provide a hardware architecture of a corresponding electronic device.

[0179] Exemplarily, Figure 9 Fig. shows a detailed architecture diagram of an electronic device 900 applicable to the present application.

[0180] As Figure 9As shown, the electronic device 900 may include a processor 910, one or more cameras 920 (the multiple displays may be represented by 1 to N, where N is a positive integer greater than 1), one or more displays 930 (the multiple displays may be represented by 1 to N), an internal memory 940, a sensor module 950, and an audio module 960, where the audio module 960 includes at least a speaker 960A and a microphone 960B.

[0181] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 900. In some other embodiments of the present application, the electronic device 900 may include more or fewer components than shown, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0182] The processor 910 may include one or more processing units. For example, the processor 910 may include an application processor (AP), a modem processor, a GPU, an image signal processor (ISP), a controller, a video codec, a DSP, a baseband processor, and / or an NPU, etc. Among them, different processing units may be independent devices or integrated in one or more processors. A memory may also be provided in the processor 910 for storing instructions and data.

[0183] The electronic device realizes the display function through the GPU, the display 930, and the application processor, etc. The GPU is a microprocessor for image processing, connected to the display 930 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 910 may include one or more GPUs, which execute program instructions to generate or change display information.

[0184] The display 930 is used to display images, videos, etc. The display 930 includes a display panel. The display panel may be a liquid crystal display (LCD). In some embodiments, the electronic device may include 1 or N displays 930, where N is a positive integer greater than 1.

[0185] In the embodiments of the present application, the electronic device displays the original images / videos, preview images / videos collected by the camera, the images / videos obtained after being processed by post-processing algorithms, and Figure 1 、 Figure 2 、 Figure 4 the ability of the user interface shown, relying on the display functions provided by the above-mentioned GPU, the display 930, and the application processor.

[0186] An electronic device can implement a shooting function through an ISP, a camera 920, a video codec, a GPU, a display screen 930, an application processor, etc. The ISP is used to process the data fed back by the camera 920.

[0187] The camera 920 is used to capture static images or videos. An object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB or YUV. In some embodiments, the electronic device may include one or N cameras 920, where N is a positive integer greater than 1.

[0188] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device selects a frequency point, the digital signal processor is used to perform a Fourier transform on the frequency point energy, etc.

[0189] The video codec is used to compress or decompress digital videos. The electronic device can support one or more video codecs. In this way, the electronic device can play or record videos in multiple coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0190] In the embodiments of the present application, for the electronic device to implement the method for processing shooting data provided in the embodiments of the present application, it first depends on the images collected by the ISP and the camera 920, and secondly also depends on the image calculation and processing capabilities provided by the video codec and the GPU. Among them, the electronic device can implement various post-processing algorithms through the computing and processing capabilities provided by the NPU.

[0191] The internal memory 940 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0192] In an embodiment of the present application, the code for implementing the method for processing captured data described in the embodiments of the present application can be stored on a non-volatile memory. When the camera application is running or when the camera application is sent to the background, the electronic device can load the executable code stored in the non-volatile memory into the random access memory.

[0193] The audio module 960 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal. The speaker 960A is used to convert an audio electrical signal into a sound signal. The electronic device can listen to music or hands-free calls through the speaker 960A. The microphone 960B is used to convert a sound signal into an electrical signal.

[0194] In an embodiment of the present application, during the process of enabling the camera to capture an image, the electronic device can simultaneously enable the microphone 960B to capture a sound signal and convert the sound signal into an electrical signal for storage. In this way, the user can obtain a video with sound.

[0195] In an embodiment of the present application, the electronic device can use the sensor module 950 (such as a touch sensor) to detect operations such as clicks and swipes by the user on the display screen 930 to implement Figure 1 , Figure 2 , Figure 4 the shooting process shown.

[0196] The term "user interface (UI)" in the specification of the present application is a media interface for interaction and information exchange between an application or an operating system and a user, and it realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is source code written in a specific computer language such as Java or Extensible Markup Language (XML). The interface source code is parsed and rendered on the terminal device and finally presented as content recognizable by the user, such as controls like pictures, text, buttons, etc. A control (also known as a widget) is a basic element of the user interface. Typical controls include a toolbar, a menubar, a text box, a button, a scrollbar, pictures, and text. The attributes and content of the controls in the interface are defined through tags or nodes. For example, XML passes through <textview> 、 <imgview> 、 <videoview>Nodes such as these are used to define the controls included in the interface. One node corresponds to one control or property in the interface, and after being parsed and rendered, the node presents as content visible to the user. In addition, in the interfaces of many applications, such as hybrid applications, there are usually also web pages included. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is source code written in a specific computer language, such as hyper text markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as content recognizable by the user by a browser or a web page display component similar to the browser function. The specific content included in the web page is also defined by tags or nodes in the web page source code. For example, HTML uses 、 、 <video> 、 <canvas>Define the elements and attributes of a web page.

[0197] A commonly used form of the user interface is the graphical user interface (GUI), which refers to the user interface related to computer operations presented in a graphical manner. It can be an interface element such as an icon, window, control, etc. displayed on the display screen of an electronic device, where the control can include visible interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, Widgets, etc.

[0198] It can be understood that if the units integrated in the above device embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above method embodiments of the present application, a computer program can be used to instruct the relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the photographing device / electronic device, recording medium, computer memory, read-only memory (ROM), RAM, electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk, or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0199] The embodiments of the present application provide a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above method embodiments.

[0200] It should be noted that in the above embodiments, the descriptions of each embodiment have their own focuses. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0201] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0202] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included within the protection scope of the present application.

[0203] The terms "first", "second", "third", "fourth" and other various term labels (if any) in the description, claims and the above drawings of the present application are used to distinguish similar objects, and do not have to be used to describe a specific order or quantity. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments described herein can be implemented in an order different from that shown or described herein.

[0204] In each embodiment of the present application, if there is no special description and logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships. The specific operation methods in the method embodiments of the present application can also be applied to the device embodiments or system embodiments.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A method for processing shooting data, applied to an electronic device, characterized in that, The method includes: In response to a shooting operation, obtaining first system information and first algorithm information, where the first system information is used to indicate the current first system resource status of the electronic device, and the first algorithm information is used to indicate the resource requirements corresponding to each algorithm in the algorithm set; Determining a target algorithm from the algorithm set according to the first system information and the first algorithm information, where the resource requirements of the target algorithm match the first system resource status; Shooting to obtain image data or video data, and processing the image data or video data according to the target algorithm.

2. The method according to claim 1, characterized in that, The method further includes: Determining scheduling priority information according to the first system information and the first algorithm information, where the scheduling priority information is used to indicate the priorities of one or more threads corresponding to the target algorithm; The processing the image data or video data obtained by shooting according to the target algorithm includes: Invoking multiple threads of the target algorithm based on the scheduling priority information to process the image data or video data.

3. The method according to claim 2, characterized in that Before invoking the multiple threads of the target algorithm according to the scheduling priority information, the method further includes: Putting the processing task corresponding to the target algorithm into a task queue; When executing the processing task of the target algorithm, selecting the multiple threads corresponding to the target algorithm from a pre-created thread pool.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Determining control parameters according to the first system information and the first algorithm information, where the control parameters include a frequency boosting parameter and a core binding parameter corresponding to the target algorithm; Configuring the control parameters as the attribute information of multiple threads corresponding to the target algorithm.

5. The method according to any one of claims 1 to 3, characterized in that, The first system resource status includes one or more of the following: processor load, remaining memory, heat generation, power consumption, battery power, sleep state; the resource requirements include one or more of the following: processor occupancy, memory occupancy, heterogeneous hardware occupancy.

6. The method according to any one of claims 1 to 3, characterized in that The first system information is further used to indicate the type of the foreground application.

7. The method according to any one of claims 1 to 3, characterized in that, Before determining the target algorithm from the algorithm set according to the first system information and the first algorithm information, the method further includes: Obtaining scene perception information, and determining the algorithm set according to the scene perception information, where the scene perception information includes one or more of the following: shooting mode, frame rate information, resolution information, lighting condition.

8. The method according to any one of claims 1 to 3, characterized in that, Before determining the target algorithm from the algorithm set according to the first system information and the first algorithm information, the method further includes: Determining that the verification of the first system information and the first algorithm information passes.

9. The method according to any one of claims 1 to 3, characterized in that, The processing the image data or video data according to the target algorithm includes: In response to an operation of switching the camera application to a background application, obtaining second system information and second algorithm information, where the second system information is used to indicate the second system resource status of the electronic device after meeting the resource scheduling requirements of the foreground application, and the second algorithm information is used to indicate the resource requirements of the algorithm to be processed in the target algorithm; Determine whether the resource requirement of the algorithm to be processed matches the second system resource status; When the resource requirement of the algorithm to be processed matches the second system resource status, use the algorithm to be processed to process the image data or video data.

10. An electronic device, characterized in that, The structure of the electronic device includes a processor and a memory; The memory is used to store a program that supports the electronic device to execute the method provided in any one of claims 1 to 9, and to store data involved in implementing the method described in any one of claims 1 to 9; The processor is configured to execute the program stored in the memory.

11. A computer-readable storage medium, characterized in that, Instructions are stored in the computer-readable storage medium, and when it runs on a computer, it causes the computer to execute the method described in any one of claims 1 to 9.

Images