Shooting method, electronic device, storage medium and program product
By estimating and controlling the operation of the shooting algorithm in electronic devices when the system load is high, the poor shooting performance problem caused by the device's inability to provide sufficient computing resources is solved, and the effect of reducing the loss rate and improving the user experience is achieved.
Patent Information
- Application Number
- CN202410350267.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-26
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-03-26
AI Technical Summary
Electronic devices cannot provide sufficient computing resources for the shooting algorithm when the system load is high, resulting in poor shooting performance and may cause picture loss.
By obtaining system load information, the time consumption of each algorithm is estimated, and whether to run the algorithm based on the shooting timeout threshold is determined to control the total running time of the shooting processing flow, and skip the algorithm that takes too long to avoid timeout.
It reduces the loss rate of image when shooting images, improves the user's shooting experience, and ensures the stability and efficiency of the shooting process.
Smart Images

Figure CN117956264B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of terminal technology, and in particular to a shooting method, electronic equipment, storage medium and program product. Background Art
[0002] With the continuous development of terminal technology, electronic devices such as mobile phones and tablets have gradually integrated functions such as communication, shooting, and audio and video, becoming an indispensable part of people's daily lives. Users can use electronic devices to capture images and record every detail of their lives. However, the computing resources of electronic devices are relatively limited. When the system load is relatively high, it is very likely that sufficient computing resources cannot be provided for the shooting algorithm, which may easily lead to poor shooting performance, resulting in the failure of the electronic device to output the captured image expected by the user, that is, the image loss phenomenon. Summary of the invention
[0003] The present application provides a shooting method, electronic device, storage medium and program product, which can reduce the image loss rate during image shooting. The technical solution is as follows:
[0004] In a first aspect, a shooting method is provided. In the method, an electronic device receives a shooting instruction; in response to the shooting instruction, executes a shooting process to obtain a target image, and the shooting process includes multiple algorithms. In the process of executing the shooting process, before the target algorithm needs to be run, the electronic device obtains system load information, determines the estimated time consumption of the target algorithm according to the system load information, and determines whether to run the target algorithm or skip the target algorithm according to the shooting timeout threshold and the estimated time consumption of the target algorithm, and the target algorithm is any one of the multiple algorithms.
[0005] The estimated time consumption of the target algorithm is the estimated time consumption of the target algorithm. The estimated time consumption of the target algorithm is used to indicate how long it takes for the target algorithm to run to completion under the system load indicated by the system load information.
[0006] The shooting timeout threshold can be preset. The shooting timeout threshold is the maximum allowed running time of the shooting process. That is, if the running time of the shooting process exceeds the shooting timeout threshold, it is determined that the shooting process has timed out and needs to be terminated.
[0007] In the present application, before running each algorithm in the shooting process, the electronic device can determine the estimated time consumption of the algorithm according to the system load information, and determine whether to run the algorithm or skip the algorithm according to the estimated time consumption and the shooting timeout threshold. In this way, the total running time of the shooting process can be controlled with the shooting timeout threshold as a constraint, so that the total running time of the shooting process is more likely to not exceed the shooting timeout threshold, thereby reducing the possibility of timeout of the shooting process, reducing the image loss rate during image shooting, and improving the user's shooting experience.
[0008] Optionally, the system load information may include operating status information of each chip in the plurality of chips.
[0009] For example, the multiple chips are chips required to execute the shooting process. For example, the multiple chips may include a chip where a CPU is located, a chip where a GPU is located, and a chip where a DSP is located.
[0010] By way of example, the operating status information of the chip includes the load rate and frequency of the chip.
[0011] In the present application, multiple chips are included in the monitoring scope, and the time consumption of the target algorithm is estimated based on the operating status information of each chip in the multiple chips, thereby improving the accuracy of the estimated time consumption.
[0012] As an optional implementation method, the operation of the electronic device determining the estimated time consumption of the target algorithm based on the system load information can be: determining the load deterioration coefficient of each chip based on the operating status information of each chip among the multiple chips; determining the estimated time consumption of the target algorithm based on the load deterioration coefficient of each chip among the multiple chips and the preset time consumption corresponding to the target algorithm.
[0013] The load deterioration coefficient of the chip is used to indicate the degree of deterioration of the current operating status of the chip (i.e., the operating status indicated by the operating status information of the chip) compared to the normal operating status of the chip (i.e., the operating status of the normal execution of the shooting process). When the chip is in the normal operating status, it can provide sufficient computing resources for the shooting process.
[0014] For example, if the load deterioration coefficient of the chip is greater than 0 and less than or equal to 1, it means that the current operating condition of the chip has not deteriorated compared to the normal operating condition; if the load deterioration coefficient of the chip is greater than 1, it means that the current operating condition of the chip has deteriorated compared to the normal operating condition.
[0015] The preset time consumption corresponding to the target algorithm can be preset. The preset time consumption corresponding to the target algorithm is the time consumption of the target algorithm when the system load is normal, that is, when sufficient computing resources are available to provide the target algorithm.
[0016] In the present application, the load deterioration coefficient of each chip among the multiple chips can reflect the degree of deterioration of the operating condition of each chip. Since the multiple chips will affect the operation of the target algorithm, the time consumption of the target algorithm can be more accurately estimated based on the load deterioration coefficient of each chip among the multiple chips and the preset time consumption corresponding to the target algorithm.
[0017] As an optional implementation method, the operation of the electronic device determining the load degradation coefficient of each chip based on the operating status information of each chip among the multiple chips can be: for the operating status information of any one of the multiple chips, the operating status information of the chip is input into the operating performance model corresponding to the chip, and the load degradation coefficient output by the operating performance model is obtained.
[0018] The operation performance model corresponding to the chip is used to determine the load deterioration coefficient of the chip according to the operation status information of the chip. In the present application, the load deterioration coefficient of the chip can be quickly obtained through the operation performance model corresponding to the chip according to the operation status information of the chip.
[0019] Each chip in the plurality of chips has its corresponding operation performance model. Optionally, the operation performance models corresponding to different chips may be different. The operation performance model corresponding to a certain chip may be obtained by training based on relevant data of the chip.
[0020] As an optional implementation method, the electronic device may determine the estimated time consumption of the target algorithm based on the load degradation coefficient of each chip in the multiple chips and the preset time consumption corresponding to the target algorithm by: taking a weighted average of the load degradation coefficient of each chip in the multiple chips according to the weight of each chip in the multiple chips to obtain the target degradation coefficient; and multiplying the target degradation coefficient by the preset time consumption corresponding to the target algorithm to obtain the estimated time consumption of the target algorithm.
[0021] Optionally, for different algorithms in the multiple algorithms, the weights of the multiple chips may be the same, or may be partially different or completely different. For example, assuming that the multiple algorithms include Algorithm 1 and Algorithm 2, and assuming that the multiple chips include Chip 1, Chip 2, and Chip 3. When determining the estimated time consumption of Algorithm 1, the weight of Chip 1 may be 0.6, the weight of Chip 2 may be 0.2, and the weight of Chip 3 may be 0.2. When determining the estimated time consumption of Algorithm 2, the weight of Chip 1 may be 0.5, the weight of Chip 2 may be 0.3, and the weight of Chip 3 may be 0.2.
[0022] For the target algorithm, the weight of any one of the multiple chips is used to indicate the importance of the chip when the target algorithm is running. If the weight of a chip is greater than the weight of another chip, it means that the importance of this chip when the target algorithm is running is higher than that of another chip when the target algorithm is running, that is, this chip plays a greater role in the process of running the target algorithm, and accordingly, this chip has a greater impact on the time consumption of the target algorithm.
[0023] In the present application, when determining the estimated time consumption of the target algorithm, not only the load deterioration coefficient of the multiple chips is considered, but also the influence of each chip in the multiple chips on the time consumption of the target algorithm is considered, so as to improve the accuracy of the estimated time consumption.
[0024] As an optional implementation method, the electronic device can determine whether to run the target algorithm or skip the target algorithm based on the shooting timeout threshold and the estimated time consumption of the target algorithm by: obtaining the target duration, where the target duration is the shooting timeout threshold minus the current running duration of the shooting processing flow; if the estimated time consumption of the target algorithm is less than or equal to the target duration, run the target algorithm; if the estimated time consumption of the target algorithm is greater than the target duration, skip the target algorithm.
[0025] After the electronic device starts executing the shooting process, it can time to determine the current running time of the shooting process, that is, the time from the start of the shooting process to the current moment. Then, the target time obtained by subtracting the current running time of the shooting process from the shooting timeout threshold is used to indicate how long the shooting process is currently allowed to run. In this case, if the next running time of the shooting process exceeds the target time, the shooting process will time out and will be terminated, which is likely to cause image loss.
[0026] For this reason, in this application, if the estimated time consumption of the target algorithm is greater than the target duration, it means that after running the target algorithm, it is very likely that the shooting process will time out. Therefore, in order to reduce the timeout probability of the shooting process, the target algorithm can be skipped in this case. If the estimated time consumption of the target algorithm is less than or equal to the target duration, it means that after running the target algorithm, it is unlikely to cause the shooting process to time out. Therefore, in this case, the target algorithm can be run normally for image processing.
[0027] It should be noted that when the target algorithm is not the last algorithm among the multiple algorithms included in the shooting processing flow, after skipping the target algorithm, or when running the target algorithm and after the target algorithm is completed, it can be determined that the next algorithm needs to be run. At this time, it can continue to determine whether to run the algorithm or skip the algorithm based on the current system load information.
[0028] In the case where the target algorithm is the last algorithm among multiple algorithms included in the shooting processing flow, if the target algorithm is skipped, the execution of the shooting processing flow is completed after the target algorithm is skipped, and the target image can be output. In this case, the output target image is an image obtained after image processing based on the algorithm run before the last algorithm; if the target algorithm is run, the execution of the shooting processing flow is completed after the target algorithm is run, and the target image can be obtained after the target algorithm is run.
[0029] In a second aspect, a photographing device is provided, which has the function of implementing the photographing method in the first aspect. The photographing device includes at least one module, and the at least one module is used to implement the photographing method provided in the first aspect.
[0030] In a third aspect, an electronic device is provided, comprising: one or more processors, and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions so that the electronic device executes the shooting method provided in the first aspect.
[0031] In a fourth aspect, a chip system is provided. The chip system is applied to an electronic device. The chip system includes one or more processors. The one or more processors are used to call computer instructions so that the electronic device executes the shooting method provided in the first aspect.
[0032] In a fifth aspect, a computer-readable storage medium is provided, which includes instructions. When the instructions are executed on an electronic device, the electronic device executes the shooting method provided in the first aspect.
[0033] In a sixth aspect, a computer program product is provided. When the computer program product is run on an electronic device, the electronic device executes the shooting method provided in the first aspect.
[0034] The technical effects obtained by the above-mentioned second, third, fourth, fifth and sixth aspects are similar to the technical effects obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;
[0036] Figure 2 is a block diagram of a software system of an electronic device provided in an embodiment of the present application;
[0037] Figure 3is a schematic diagram of an interface of a shooting process provided in an embodiment of the present application;
[0038] Figure 4 is a flow chart of a shooting method provided by an embodiment of the present application;
[0039] Figure 5 is a flow chart of a process for determining estimated time consumption provided by an embodiment of the present application;
[0040] Figure 6 is a schematic diagram of a shooting method provided in an embodiment of the present application;
[0041] Figure 7 is a flow chart of another shooting method provided by an embodiment of the present application;
[0042] Figure 8 It is a structural schematic diagram of a shooting device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0043] In the following description, specific details such as specific system structures and technologies are provided for illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details.
[0044] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or their collections. The terms "comprising", "including", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.
[0045] It should be understood that the "one or more" mentioned in this application refers to one, two or more than two, and the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B. The "and / or" in this article is just a way to describe the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.
[0046] In order to clearly describe the technical solution of the present application, the words "first", "second" and the like are used to distinguish the same or similar items with substantially the same functions and effects. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not necessarily limit the difference.
[0047] The phrases such as "one embodiment" or "some embodiments" described in the present application mean that the specific features, structures or characteristics described in the embodiment are included in one or more embodiments of the present application. Therefore, the phrases such as "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments" etc. that appear in different places in the present application do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.
[0048] The shooting method provided in the embodiment of the present application can be applied to electronic devices. For example, the electronic device can be a mobile terminal (MT), a mobile station (MS), a mobile unit (MU), a wireless unit, a remote unit, a user agent, a mobile client, etc. For example, the electronic device can be a mobile phone, a smart screen device, a tablet computer, a wearable device, a digital camera, a vehicle-mounted device, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a laptop computer, etc., and the embodiment of the present application does not limit this.
[0049] Figure 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 1The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
[0050] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0051] The processor 110 may include one or more processing units, for example, the processor 110 may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.
[0052] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.
[0053] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves system efficiency.
[0054] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing files such as music and videos in the external memory card.
[0055] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data (such as audio data, a phone book, etc.) created by the electronic device 100 during use. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
[0056] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, the baseband processor, and the like.
[0057] The mobile communication module 150 can provide wireless communication solutions including second generation mobile communication technology (2G) / third generation mobile communication technology (3G) / fourth generation mobile communication technology (4G) / fifth generation mobile communication technology (5G) applied on the electronic device 100.
[0058] The wireless communication module 160 can provide wireless communication solutions applied to the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc.
[0059] The electronic device 100 can implement audio functions, such as music playback and recording, through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.
[0060] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.
[0061] The electronic device 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor.
[0062] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 193.
[0063] The camera 193 is used to capture static images or videos. For example, the camera 193 can be triggered to start by an application program instruction to realize the shooting function, such as shooting and acquiring images of any scene. The camera 193 may include components such as an imaging lens, a filter, and an image sensor. The light emitted or reflected by the object enters the imaging lens, passes through the filter, and finally converges on the image sensor. The imaging lens is mainly used to converge the light emitted or reflected by all objects in the shooting angle; the filter is mainly used to filter out the redundant light waves in the light; the image sensor can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The image sensor is mainly used to convert the received light signal into an electrical signal, and then transmit the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as red green blue (RGB), luminance chrominance (YUV), etc.
[0064] The electronic device 100 can realize the display function through a GPU, a display screen 194, an application processor, and the like.
[0065] The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.
[0066] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel.
[0067] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor 180K can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.
[0068] Next, the software system of the electronic device 100 will be described.
[0069] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software system of the electronic device 100.
[0070] Figure 2 1 is a block diagram of a software system of an electronic device 100 provided in an embodiment of the present application. Figure 2 The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into an application layer, an application framework layer, an Android runtime (AndroidRuntime), a system layer, and a kernel layer.
[0071] The application layer can include a series of applications. Figure 2 As shown, the application layer may include applications such as camera, calendar, map, WLAN, music, short message, gallery, call, navigation, Bluetooth, video, etc.
[0072] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.
[0073] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, a time estimation module, and the like.
[0074] The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make these data accessible to applications. These data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build the display interface of the application. The display interface can be composed of one or more views, for example, including a view that displays a text notification icon, including a view that displays text, and including a view that displays pictures. The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connected, hung up, etc.). The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the top status bar of the system in the form of a chart or scroll bar text, such as a notification of an application running in the background. The notification manager can also be a notification that appears on the screen in the form of a dialog window, such as a text message prompt in the status bar, a prompt sound, an electronic device vibrating, an indicator light flashing, etc. The time consumption estimation module is used to estimate the time consumption of each of the multiple algorithms in the shooting process.
[0075] Android Runtime includes core libraries and virtual machines. Android Runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one is the function that the Java language needs to call, and the other is the Android core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection.
[0076] The system layer can include multiple functional modules, such as: surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc. The surface manager is used to manage the display subsystem and provide the fusion of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications. The media library supports the playback and recording of multiple common audio and video formats, as well as static image files. The media library can support multiple audio and video encoding formats, such as: MPEG4, H.264 (also known as advanced video coding (AVC)), moving picture experts group audio layer III (MP3), advanced audio coding (AAC), adaptive multi-rate (AMR), joint photographic experts group (JPG), portable network graphics (PNG), etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing. A 2D graphics engine is a drawing engine for 2D drawings.
[0077] The kernel layer is the layer between hardware and software. The kernel layer may include display driver, camera driver, audio driver, sensor driver, etc.
[0078] The embodiments of the present application do not particularly limit the specific structure of the execution subject of the shooting method, as long as the code recording the shooting method provided by the embodiments of the present application can be run to perform processing according to the shooting method provided by the embodiments of the present application. For example, the execution subject of the shooting method provided by the embodiments of the present application can be a functional module in an electronic device that can call and execute a program, or can be a processing device applied to an electronic device, such as a chip.
[0079] Next, the application scenarios involved in the embodiments of the present application are explained.
[0080] Currently, users often use electronic devices such as mobile phones and tablets to capture images and record every detail of their lives.
[0081] For example, Figure 3As shown in FIG. 1 (a), a desktop 310 displayed by the electronic device displays an icon 311 of a camera application. If the user wants to take a picture, the user can click the icon 311 of the camera application. In response to the click operation, Figure 3 As shown in Figure (b) in the figure, the electronic device starts the camera application and displays the camera application's shooting interface 320, which displays a preview window 321, a shooting mode control 322, a shooting control 323, etc. If the user determines that the preview image displayed in the preview window 321 meets his needs, he can click the shooting control 323 to shoot. In response to the click operation, the electronic device can shoot an image, obtain a target image, and save the target image to the gallery application. Figure 3 As shown in FIG. 3 (c), after obtaining the target image, the electronic device may also display a thumbnail 324 of the target image on the shooting interface 320. If the user wants to view the target image, the user may click on the thumbnail 324 of the target image. In response to the click operation, Figure 3 As shown in Figure (d), the electronic device can display the target image.
[0082] After the user clicks the shooting control 323, the electronic device shoots an image, and the process of obtaining the target image can be called a shooting process flow, or a shooting pipeline. The shooting process flow may refer to a series of processing stages from the light entering the camera to the generation of the final image during shooting. Specifically, the shooting process flow may include multiple algorithms, which are used to process the raw image data collected by the camera (including but not limited to the front camera, rear camera, etc.) to obtain the final target image. By way of example, the multiple algorithms may include a basic shooting algorithm, a blurring algorithm, a skin beautification algorithm, and the like.
[0083] The camera application may include multiple shooting modes, such as a large aperture mode, a night scene mode, a portrait mode, a photo mode, a video mode, a professional mode, or other modes. The shooting process flow of each shooting mode in the multiple shooting modes may be pre-set. In some embodiments, the shooting process flows of different shooting modes may be different, that is, some or all of the multiple algorithms included in the shooting process flows of different shooting modes may be different. For example, the shooting process flow of the portrait mode includes a blur algorithm, while the shooting process flow of the ordinary photo mode does not include a blur algorithm.
[0084] The electronic device is pre-set with a shooting timeout threshold. After the user clicks the shooting control 323, the electronic device starts to execute the shooting process and synchronizes the timing. In some cases, if the shooting process takes longer than the shooting timeout threshold, that is, the shooting process is not completed before the timeout, the electronic device will terminate the shooting process. In this case, the image shooting fails, and the electronic device cannot output the shot image, that is, the image loss phenomenon occurs.
[0085] At present, the computing resources of electronic devices such as mobile phones and tablets are relatively limited, and it is easy to have high system load. In the case of high system load, if the user uses an electronic device to shoot, the electronic device is likely to be unable to provide sufficient computing resources for the shooting process, which is prone to poor shooting performance, resulting in a long shooting process or even timeout resulting in lost images, which seriously affects the user's shooting experience.
[0086] To this end, an embodiment of the present application provides a shooting method, which can determine the estimated time consumption of each of the multiple algorithms included in the shooting processing flow according to the system load information, and then selectively skip some of the multiple algorithms according to the estimated time consumption of each algorithm, so as to reduce the possibility of timeout of the shooting processing flow when the system load is high, thereby reducing the image loss rate during image shooting and improving the user's shooting experience.
[0087] Next, the shooting method provided in the embodiment of the present application is explained in detail.
[0088] Figure 4 is a flow chart of a shooting method provided by an embodiment of the present application. The method can be Figure 1 to Figure 2 The electronic device described in the embodiment is executed. Figure 4 , the method comprises the following steps:
[0089] Step 401: The electronic device receives an operation instructing to start a camera application.
[0090] Optionally, the operation of instructing to launch the camera application may be triggered by a user.
[0091] For example, when the electronic device displays the desktop, it can receive a click operation on the icon of the camera application displayed on the desktop, and the click operation is used to instruct to start the camera application. In response to the click operation, the electronic device can start the camera application.
[0092] For example, when the electronic device is in a locked screen state, it can receive a right-slide operation on the screen, and the right-slide operation is used to instruct to start the camera application. In response to the right-slide operation, the electronic device can start the camera application.
[0093] For example, when the electronic device is in a locked screen state, the electronic device may display an icon of a camera application in the lock screen interface, and the electronic device may receive a click operation on the icon of the camera application displayed in the lock screen interface, the click operation being used to instruct to start the camera application. In response to the click operation, the electronic device may start the camera application.
[0094] Alternatively, when the electronic device is running other applications, if the application has the permission to call the camera application, the electronic device can receive a click operation on the corresponding control in the application, and the click operation is used to instruct to start the camera application. In response to the click operation, the electronic device can start the camera application. For example, if the electronic device is running an instant messaging application, the user can click on the control of the camera function in the application interface of the instant messaging application, and in response to the click operation on the control of the camera function, the electronic device can start the camera application.
[0095] Of course, not limited to the above examples, the electronic device may also receive an operation instructing to start a camera application in other situations, and then start the camera application.
[0096] Step 402: The electronic device starts a camera application to display a shooting interface of the camera application.
[0097] After the electronic device starts the camera application, the camera application starts to run and can display a shooting interface during the running.
[0098] The shooting interface may display a preview window, a shooting mode control, a shooting control, etc. Among them, the preview window is used to display a preview image under the current shooting angle of the electronic device. The number of the shooting mode controls can be multiple, and is used to provide different shooting modes, such as large aperture, night scene, portrait, photo, video, professional and other modes. The user can use the shooting mode control to switch the shooting mode currently used by the camera application. The shooting control is used to indicate image shooting.
[0099] For example, users can Figure 3 In the desktop 310 shown in FIG. (a), the camera application icon 311 is clicked. In response to the click operation, as shown in FIG. Figure 3 As shown in FIG. 2( b ), the electronic device may start a camera application to display a shooting interface 320 of the camera application. The shooting interface 320 displays a preview window 321 , a shooting mode control 322 , a shooting control 323 , and the like.
[0100] Step 403: The camera application receives an operation acting on a shooting control in the shooting interface.
[0101] The operation acting on the shooting control here is a triggering operation on the shooting control, such as a click operation, a gesture operation, a voice operation, a body sensing operation, etc., to instruct image shooting.
[0102] For example, users can Figure 3 In the shooting interface 320 shown in FIG. 7 (b), the shooting control 323 is clicked to instruct image shooting.
[0103] Step 404: In response to the operation on the shooting control, the camera application determines a plurality of algorithms included in the shooting processing flow.
[0104] The shooting process flow is a shooting process flow of a shooting mode currently adopted by the camera application.
[0105] The multiple algorithms in the shooting process are executed in sequence, and the execution order of the multiple algorithms is preset. For example, the multiple algorithms may include algorithms executed in series or algorithms executed in parallel, which is not limited in the embodiments of the present application.
[0106] For example, the multiple algorithms may include Algorithm 1, Algorithm 2, and Algorithm 3. Algorithm 1, Algorithm 2, and Algorithm 3 are executed in series. That is, Algorithm 1 is run first, and after Algorithm 1 is run, Algorithm 2 is run, and after Algorithm 2 is run, Algorithm 3 is run, and after Algorithm 3 is run, the target image is obtained.
[0107] For another example, the multiple algorithms may include Algorithm 4, Algorithm 5, Algorithm 6, Algorithm 7, and Algorithm 8. Algorithm 4, Algorithm 5, Algorithm 7, and Algorithm 8 are executed in series, and Algorithm 4, Algorithm 6, Algorithm 7, and Algorithm 8 are executed in series, and Algorithm 5 and Algorithm 6 are executed in parallel. That is, Algorithm 4 is executed first, and after Algorithm 4 is executed, Algorithm 5 and Algorithm 6 are executed, and after Algorithm 5 and Algorithm 6 are executed, Algorithm 7 is executed, and after Algorithm 7 is executed, Algorithm 8 is executed, and after Algorithm 8 is executed, the target image is obtained.
[0108] Assume that the multiple algorithms in the shooting process include a basic shooting algorithm and a blurring algorithm, and the basic shooting algorithm and the blurring algorithm are executed in series. This is taken as an example for exemplary description.
[0109] Step 405: Before the basic photographing algorithm needs to be run, the camera application calls the time consumption estimation module to determine the estimated time consumption of the basic photographing algorithm.
[0110] The basic photographing algorithm is used to process the input image data, such as noise reduction, color adjustment, brightness adjustment, etc. For example, when the basic photographing algorithm is the first algorithm in the photographing process, the image data input to the basic photographing algorithm may be the original image data captured by the camera.
[0111] The estimated time consumption of the basic photographing algorithm is the estimated possible time consumption of the basic photographing algorithm under the current load condition of the electronic device, and the time consumption refers to the time length from the start of the basic photographing algorithm to the completion of the operation.
[0112] After receiving the operation acting on the shooting control, the camera application can start to execute the shooting process, that is, it can run multiple algorithms in the shooting process in sequence. In this case, before the camera application needs to run an algorithm in the shooting process, it can first call the time estimation module to determine the estimated time of the algorithm, and then determine whether to run the algorithm or skip (i.e. bypass) the algorithm based on the estimated time.
[0113] In some embodiments, see Figure 5 The operation of the time estimation module determining the estimated time consumption of the basic photographing algorithm may include the following steps 4051 to 4053.
[0114] Step 4051: The time consumption estimation module obtains system load information.
[0115] The system load information is used to indicate the system load condition. By way of example, the system load information may include operating status information of each chip in the plurality of chips.
[0116] Optionally, the multiple chips may be chips required to execute the shooting process. For example, the multiple chips may include a chip where a CPU is located, a chip where a GPU is located, a chip where a DSP is located, etc., which is not limited in the present embodiment of the application.
[0117] The operating status information of any one of the multiple chips may include information such as the load rate and frequency of the chip. Of course, it may also include other information that can reflect the operating status of the chip, which is not limited in the embodiments of the present application.
[0118] In some embodiments, for any one chip among the multiple chips, the time consumption estimation module may obtain the operating status information of the chip through the monitoring interface of the chip or related system tools.
[0119] Step 4052: The time consumption estimation module determines the load degradation coefficient according to the system load information.
[0120] The load deterioration coefficient is used to indicate the degree of deterioration of the current system load of the electronic device (i.e., the system load indicated by the system load information) compared to the normal system load (i.e., the system load that can normally execute the shooting process). In the normal system load, the electronic device can provide sufficient computing resources for the shooting process.
[0121] For example, if the load deterioration coefficient is greater than 0 and less than or equal to 1, it means that the current system load condition has not deteriorated compared with the normal system load condition, that is, the current system load is normal; if the load deterioration coefficient is greater than 1, it means that the current system load condition has deteriorated compared with the normal system load condition, that is, the current system load is high.
[0122] In some embodiments, the operation of step 4052 may be: for the operating status information of any one of the multiple chips, the time consumption estimation module inputs the operating status information of the chip into the operating performance model corresponding to the chip to obtain the load deterioration coefficient output by the operating performance model.
[0123] The operation performance model corresponding to the chip is used to determine the load deterioration coefficient of the chip according to the operation status information of the chip. The load deterioration coefficient of the chip is used to indicate the degree of deterioration of the current operation status of the chip (i.e., the operation status indicated by the operation status information of the chip) compared with the normal operation status of the chip (i.e., the operation status of the normal execution of the shooting process). When the chip is in the normal operation status, it can provide sufficient computing resources for the shooting process.
[0124] For example, if the load deterioration coefficient of the chip is greater than 0 and less than or equal to 1, it means that the current operating condition of the chip has not deteriorated compared to the normal operating condition; if the load deterioration coefficient of the chip is greater than 1, it means that the current operating condition of the chip has deteriorated compared to the normal operating condition.
[0125] Each chip in the plurality of chips has its corresponding operation performance model. Optionally, the operation performance models corresponding to different chips may be different. The operation performance model corresponding to a certain chip may be obtained by training based on relevant data of the chip.
[0126] The operation performance model may be a neural network model, such as a lightweight neural network model. For example, the operation performance model may be a convolutional neural network model, a recurrent neural network model, etc., which is not limited in the present embodiment of the application.
[0127] For example, the operation performance model may be obtained by training the electronic device, or the operation performance model may be obtained by training other devices, and the other devices may send the trained operation performance model to the electronic device for use.
[0128] For example, the operation of training to obtain the operating performance model may be: obtaining a plurality of training samples, and training a neural network model using the plurality of training samples to obtain the operating performance model.
[0129] For example, the neural network model may include multiple network layers, which may include an input layer, multiple hidden layers, and an output layer. The input layer is responsible for receiving input data; the output layer is responsible for outputting processed data; multiple hidden layers are located between the input layer and the output layer, responsible for processing data, and the multiple hidden layers are invisible to the outside.
[0130] The plurality of training samples may be pre-set. Each of the plurality of training samples includes input data and a sample tag, the input data is the operating status information of the chip, and the sample tag is the load deterioration coefficient corresponding to the operating status information.
[0131] In some embodiments, the multiple training samples can be collected offline by a technician. For example, the technician can run a shooting algorithm (including but not limited to any one of the multiple algorithms included in the above-mentioned shooting process flow) through the chip when the chip is in normal operating conditions to obtain a first time consumption of the shooting algorithm. Then control the chip to be in different operating conditions. For any operating condition, the operating condition information of the chip is collected when the chip is in this operating condition, and the shooting algorithm is run through the chip to obtain a second time consumption of the shooting algorithm, and the second time consumption is divided by the first time consumption to obtain a load deterioration coefficient corresponding to the collected operating condition information.
[0132] For example, when the neural network model is trained using the multiple training samples, for any one of the multiple training samples, the input data in the training sample can be input into the neural network model to obtain output data; the loss value between the output data and the sample label in the training sample is determined by the loss function; and the parameters in the neural network model are adjusted according to the loss value. After the parameters in the neural network model are adjusted based on each of the multiple training samples, the neural network model with the adjusted parameters is the operating performance model. Among them, the operation of adjusting the parameters in the neural network model according to the loss value can refer to the relevant technology, and the embodiments of the present application will not be elaborated in detail.
[0133] Step 4053: The time consumption estimation module determines the estimated time consumption of the basic photographing algorithm according to the load deterioration coefficient.
[0134] Optionally, each chip in the plurality of chips has a weight, and the weight of each chip in the plurality of chips can be preset. For example, the weight of each chip in the plurality of chips is greater than or equal to 0 and less than or equal to 1, and the sum of the weights of the plurality of chips is 1.
[0135] In some embodiments, for different algorithms in the multiple algorithms, the weights of the multiple chips may be the same, or may be partially different or completely different. For example, assume that the multiple chips include chip 1, chip 2, and chip 3. When determining the estimated time consumption of the basic photography algorithm, the weight of chip 1 may be 0.6, the weight of chip 2 may be 0.2, and the weight of chip 3 may be 0.2. When determining the estimated time consumption of the blurring algorithm, the weight of chip 1 may be 0.5, the weight of chip 2 may be 0.3, and the weight of chip 3 may be 0.2.
[0136] For the basic photographing algorithm, the weight of any one of the multiple chips is used to indicate the importance of the chip when the basic photographing algorithm is running. If the weight of a chip is greater than the weight of another chip, it means that the importance of this chip when the basic photographing algorithm is running is higher than the importance of another chip when the basic photographing algorithm is running, that is, this chip plays a greater role in the process of running the basic photographing algorithm, and accordingly, this chip has a greater impact on the time consumption of the basic photographing algorithm.
[0137] In some embodiments, the operation of step 4053 may be: the time estimation module performs weighted averaging of the load degradation coefficient of each chip in the multiple chips according to the weight of each chip in the multiple chips to obtain a target degradation coefficient; and multiplies the target degradation coefficient by the preset time corresponding to the basic photographing algorithm to obtain the estimated time of the basic photographing algorithm.
[0138] The preset time consumption corresponding to the basic photographing algorithm can be preset. The preset time consumption corresponding to the basic photographing algorithm is the time consumption of the basic photographing algorithm when the system load is normal, that is, when there are sufficient computing resources available to the basic photographing algorithm.
[0139] It should be noted that the embodiment of the present application includes multiple chips in the monitoring scope, and estimates the time consumption of the basic photography algorithm according to the operating status information of each chip in the multiple chips, thereby improving the accuracy of the estimated time consumption.
[0140] Step 406: The camera application determines whether the estimated time consumption of the basic photographing algorithm is greater than the target time consumption, where the target time consumption is obtained by subtracting the current running time consumption of the photographing processing flow from the photographing timeout threshold. If the estimated time consumption of the basic photographing algorithm is greater than the target time consumption, step 407 is executed; if the estimated time consumption of the basic photographing algorithm is less than or equal to the target time consumption, step 408 is executed.
[0141] The shooting timeout threshold can be preset. The shooting timeout threshold is the maximum allowed running time of the shooting process. That is, when the running time of the shooting process is greater than the shooting timeout threshold, it is determined that the shooting process has timed out, and the camera application can terminate the shooting process.
[0142] After the camera application starts executing the shooting process, it can count the time to determine the current running time of the shooting process, that is, the time from the start of the shooting process to the current moment. Then, the target time obtained by subtracting the current running time of the shooting process from the shooting timeout threshold is used to indicate how long the shooting process is currently allowed to run. In this case, if the next running time of the shooting process exceeds the target time, the shooting process will time out and will be terminated, which is likely to cause image loss.
[0143] Step 407: When the estimated time consumption of the basic photographing algorithm is greater than the target time, the camera application skips the basic photographing algorithm and determines that the next algorithm, namely the blurring algorithm, needs to be run.
[0144] If the estimated time consumption of the basic photographing algorithm is greater than the target time length, it means that after running the basic photographing algorithm, it is very likely that the photographing process will time out. Therefore, in order to reduce the timeout probability of the photographing process, the embodiment of the present application can skip the basic photographing algorithm in this case.
[0145] Step 408: The camera application runs the basic photographing algorithm when the estimated time taken by the basic photographing algorithm is less than or equal to the target time. After the basic photographing algorithm is completed, it is determined that the next algorithm, namely the blurring algorithm, needs to be run.
[0146] If the estimated time consumption of the basic photographing algorithm is less than or equal to the target time length, it means that after running the basic photographing algorithm, it is unlikely to cause the photographing processing flow to time out. Therefore, the embodiment of the present application can normally run the basic photographing algorithm to perform image processing in this case.
[0147] Step 409: before the blur algorithm needs to be run, the camera application calls the time consumption estimation module to determine the estimated time consumption of the blur algorithm.
[0148] For example, the blur algorithm is used to process the input image data, such as background blur, etc. For example, when the blur algorithm is the next algorithm of the basic photographing algorithm, if the basic photographing algorithm is skipped before, the image data input to the blur algorithm can be the original image data collected by the camera; if the basic photographing algorithm is not skipped before, but the basic photographing algorithm is run, the image data input to the blur algorithm can be the image data obtained after image processing based on the basic photographing algorithm.
[0149] The estimated time consumption of the de-emulation algorithm is the estimated possible time consumption of the de-emulation algorithm under the current load condition of the electronic device, and the time consumption refers to the time length from the start of the de-emulation algorithm to the completion of the operation.
[0150] In some embodiments, the operation of the time estimation module determining the estimated time consumption of the blurring algorithm may include the following steps (1) to (3).
[0151] (1) The time estimation module obtains system load information.
[0152] The operation of step (1) is similar to the operation of step 4051 above, and will not be described in detail in this embodiment of the present application.
[0153] (2) The time consumption estimation module determines the load deterioration coefficient based on the system load information.
[0154] The operation of step (2) is similar to the operation of the above-mentioned step 4052, and will not be described in detail in this embodiment of the present application.
[0155] (3) The time consumption estimation module determines the estimated time consumption of the virtualization algorithm according to the load deterioration coefficient.
[0156] For the defocusing algorithm, the weight of any one of the multiple chips is used to indicate the importance of the chip when the defocusing algorithm is running. If the weight of one chip is greater than the weight of another chip, it means that the importance of this chip when the defocusing algorithm is running is higher than the importance of another chip when the defocusing algorithm is running, that is, this chip plays a greater role in the process of running the defocusing algorithm, and accordingly, this chip has a greater impact on the time consumption of the defocusing algorithm.
[0157] In some embodiments, the operation of step (3) may be: the time consumption estimation module performs a weighted average of the load degradation coefficient of each chip in the multiple chips according to the weight of each chip in the multiple chips to obtain a target degradation coefficient; and multiplies the target degradation coefficient by the preset time consumption corresponding to the virtualization algorithm to obtain the estimated time consumption of the virtualization algorithm.
[0158] The preset time consumption corresponding to the defocusing algorithm can be preset. The preset time consumption corresponding to the defocusing algorithm is the time consumption of the defocusing algorithm when the system load is normal, that is, when there are sufficient computing resources available to provide the defocusing algorithm.
[0159] It should be noted that the embodiment of the present application includes multiple chips in the monitoring scope, and estimates the time consumption of the virtualization algorithm based on the operating status information of each chip in the multiple chips, thereby improving the accuracy of the estimated time consumption.
[0160] Step 410: The camera application determines whether the estimated time of the blur algorithm is greater than the target time, where the target time is the shooting timeout threshold minus the current running time of the shooting process; if the estimated time of the blur algorithm is greater than the target time, execute step 411; if the estimated time of the blur algorithm is less than or equal to the target time, execute step 412.
[0161] The operation of step 410 is similar to the operation of the above-mentioned step 406, and will not be described in detail in the embodiment of the present application.
[0162] Step 411: When the estimated time required by the blurring algorithm is greater than the target time, the camera application skips the blurring algorithm and outputs the target image.
[0163] If the estimated time consumption of the blur algorithm is greater than the target time, it means that after running the blur algorithm, it is very likely that the shooting process will time out. Therefore, in order to reduce the timeout probability of the shooting process, the embodiment of the present application can skip the blur algorithm in this case.
[0164] For example, when the blurring algorithm is the last algorithm among multiple algorithms included in the shooting processing flow, after skipping the blurring algorithm, the execution of the shooting processing flow is completed, and the target image can be output. In this case, the output target image is an image obtained after image processing based on the algorithm run before the blurring algorithm.
[0165] Optionally, the camera application can save the target image to the gallery application. In some embodiments, the camera application can also display a thumbnail of the target image in the shooting interface.
[0166] Step 412: When the estimated duration of the blurring algorithm is less than or equal to the target duration, the camera application runs the blurring algorithm to obtain the target image.
[0167] If the estimated time consumption of the blur algorithm is less than or equal to the target time, it means that after running the blur algorithm, it is unlikely to cause the shooting process to time out. Therefore, the embodiment of the present application can normally run the blur algorithm to perform image processing in this case. For example, when the blur algorithm is the last algorithm among the multiple algorithms included in the shooting process, the blur algorithm is run to process the input image data, and the execution of the shooting process is completed after the blur algorithm is run, and the target image can be obtained.
[0168] Optionally, the camera application can save the target image to the gallery application. In some embodiments, the camera application can also display a thumbnail of the target image in the shooting interface.
[0169] In an embodiment of the present application, after receiving an operation acting on a shooting control, the electronic device can execute a shooting process to obtain a target image. In the process of executing the shooting process, for any algorithm in the shooting process, the electronic device can determine the estimated time of the algorithm based on the system load information before running the algorithm, and then determine whether to run the algorithm or skip the algorithm based on the shooting timeout threshold and the estimated time of the algorithm. In this way, the total running time of the shooting process can be controlled with the shooting timeout threshold as a constraint, so that the total running time of the shooting process is more likely to not exceed the shooting timeout threshold, thereby reducing the possibility of timeout of the shooting process, reducing the image loss rate during image shooting, and improving the user's shooting experience.
[0170] In addition, in the embodiment of the present application, corresponding time estimation logic is deployed for each algorithm in the shooting process, and the time estimation logic of each algorithm is independent of each other, highly flexible, and highly portable. Moreover, in the embodiment of the present application, it is possible to determine whether to skip any algorithm based on the estimated time of any algorithm, so as to run as many algorithms in the shooting process as possible and skip as few algorithms in the shooting process as possible while reducing the timeout probability of the shooting process, so as to improve the image quality of the target image finally obtained.
[0171] Below Figure 6 For example, the above Figure 4 The shooting method provided in the embodiment is exemplified for illustration.
[0172] like Figure 6 As shown, after the user clicks the shooting control, the electronic device executes the shooting process, which includes a basic shooting algorithm and a blurring algorithm. Assume that the execution of the shooting process requires the use of chips 1, 2, and 3.
[0173] In the process of executing the shooting process, before running the basic shooting algorithm, according to the running status information of chip 1, the load deterioration coefficient of chip 1 is obtained through the running performance model corresponding to chip 1; according to the running status information of chip 2, the load deterioration coefficient of chip 2 is obtained through the running performance model corresponding to chip 2; according to the running status information of chip 3, the load deterioration coefficient of chip 3 is obtained through the running performance model corresponding to chip 3. Afterwards, according to the weight of chip 1, the weight of chip 2, the weight of chip 3, the load deterioration coefficient of chip 1, the load deterioration coefficient of chip 2, the load deterioration coefficient of chip 3 and the preset time corresponding to the basic shooting algorithm, the estimated time consumption of the basic shooting algorithm is determined. If the estimated time consumption is greater than the target time length (i.e., the time length obtained by subtracting the current running time length of the shooting process from the shooting timeout threshold), the basic shooting algorithm is skipped; otherwise, the basic shooting algorithm is run.
[0174] Afterwards, before running the blur algorithm, the estimated time consumption of the blur algorithm is determined through operations similar to the above. If the estimated time consumption is greater than the target time length, the blur algorithm is skipped and the target image is output; otherwise, the blur algorithm is run to obtain the target image.
[0175] Figure 7 is a flow chart of a shooting method provided by an embodiment of the present application. The method can be Figure 1 to Figure 2 The electronic device described in the embodiment is executed. Figure 7 , the method comprises the following steps:
[0176] Step 701: The electronic device receives a shooting instruction.
[0177] The shooting instruction is used to instruct to capture an image. The shooting instruction can be triggered by the user. For example, the user can click the shooting control in the shooting interface of the camera application to trigger the shooting instruction. Alternatively, when the electronic device displays the shooting interface of the camera application, the user can also trigger the shooting instruction through voice operation, gesture operation, somatosensory operation, etc., which is not limited in the embodiments of the present application.
[0178] Step 702: In response to the shooting instruction, the electronic device executes a shooting process to obtain a target image.
[0179] The shooting process includes a plurality of algorithms. The shooting process has been explained in the above embodiment and will not be described again here.
[0180] The target image is an image obtained by shooting. The electronic device can save the target image to a gallery application. In some embodiments, the electronic device can also display a thumbnail of the target image on the shooting interface.
[0181] In the process of executing the shooting process, the electronic device can execute the following steps 7021 to 7023.
[0182] Step 7021: The electronic device obtains system load information before running a target algorithm, where the target algorithm is any one of the multiple algorithms.
[0183] The operation of the electronic device obtaining the system load information is similar to the operation of the time consumption estimation module obtaining the system load information in the above step 4051, and will not be described in detail in this embodiment of the present application.
[0184] Step 7022: The electronic device determines the estimated time required for the target algorithm based on the system load information.
[0185] The estimated time consumption of the target algorithm is the estimated time consumption of the target algorithm. The estimated time consumption of the target algorithm is used to indicate how long it takes for the target algorithm to run to completion under the system load indicated by the system load information.
[0186] In some embodiments, the operation of step 7022 may be: the electronic device determines the load deterioration coefficient of each chip according to the operating status information of each chip in the multiple chips, and determines the estimated time consumption of the target algorithm according to the load deterioration coefficient of each chip in the multiple chips and the preset time consumption corresponding to the target algorithm.
[0187] The operation of the electronic device determining the load deterioration coefficient of each chip according to the operating status information of each chip in the multiple chips can refer to the above-mentioned step 4052, which will not be repeated here.
[0188] The operation of the electronic device determining the estimated time consumption of the target algorithm based on the load deterioration coefficient of each chip in the multiple chips and the preset time consumption corresponding to the target algorithm can refer to the above-mentioned step 4053 and will not be repeated here.
[0189] Step 7023: The electronic device determines whether to run the target algorithm or skip the target algorithm based on the shooting timeout threshold and the estimated time consumption of the target algorithm.
[0190] In the embodiment of the present application, before running each algorithm in the shooting process, the electronic device can determine the estimated time consumption of the algorithm according to the system load information, and determine whether to run the algorithm or skip the algorithm according to the estimated time consumption and the shooting timeout threshold. In this way, the total running time of the shooting process can be controlled with the shooting timeout threshold as a constraint, so that the total running time of the shooting process is more likely to not exceed the shooting timeout threshold, thereby reducing the possibility of timeout of the shooting process, reducing the image loss rate during image shooting, and improving the user's shooting experience.
[0191] In some embodiments, the operation of step 7023 may be: obtaining a target duration, which is obtained by subtracting the current running duration of the shooting process from the shooting timeout threshold; if the estimated time of the target algorithm is less than or equal to the target duration, running the target algorithm; if the estimated time of the target algorithm is greater than the target duration, skipping the target algorithm. For specific operations, refer to the above steps 406 to 408, or refer to the above steps 410 to 412, which will not be repeated here.
[0192] It should be noted that when the target algorithm is not the last algorithm among the multiple algorithms included in the shooting processing flow, after skipping the target algorithm, or when running the target algorithm and after the target algorithm is completed, it can be determined that the next algorithm needs to be run. At this time, the algorithm can be used as the target algorithm, and then the above steps 7021 to 7023 can be followed to determine whether to run the algorithm or skip the algorithm.
[0193] In the case where the target algorithm is the last algorithm among multiple algorithms included in the shooting processing flow, if the target algorithm is skipped, the execution of the shooting processing flow is completed after the target algorithm is skipped, and the target image can be output. In this case, the output target image is an image obtained after image processing based on the algorithm run before the last algorithm; if the target algorithm is run, the execution of the shooting processing flow is completed after the target algorithm is run, and the target image can be obtained after the target algorithm is run.
[0194] It should be noted that the embodiments of the present application can be applied not only to shooting scenarios, but also to other scenarios where the running speed needs to be considered and the algorithm operation can be dynamically adjusted.
[0195] For example, in some scenarios, an electronic device needs to execute a target processing flow, and the target processing flow is used to complete a specific task, and the target processing flow includes multiple algorithms for completing this task, and any one or more of the multiple algorithms can be skipped, that is, skipping some of the multiple algorithms in the process of executing the target processing flow will not affect the output of the result. In this case, before the electronic device needs to run any one of the multiple algorithms included in the target processing flow, it can determine the estimated time consumption of the algorithm according to the current system load information, and determine whether to run the algorithm or skip the algorithm according to the timeout threshold corresponding to the target processing flow and the estimated time consumption of the algorithm. In this way, the total running time of the target processing flow can be controlled with the timeout threshold corresponding to the target processing flow as a constraint, so that the total running time of the target processing flow is more likely to not exceed the timeout threshold, thereby reducing the possibility of timeout of the target processing flow and improving the execution success rate of the target processing flow.
[0196] Figure 8is a schematic diagram of a photographing device provided in an embodiment of the present application. The device can be implemented as part or all of a computer device by software, hardware, or a combination of both. The computer device can be the above Figure 1 to Figure 2 The electronic device described in the embodiment. Figure 8 The device includes: a receiving module 801 and an execution module 802.
[0197] Receiving module 801, used for receiving shooting instructions;
[0198] The execution module 802 is used to respond to the shooting instruction, execute the shooting processing flow, and obtain the target image. The shooting processing flow includes multiple algorithms; wherein, in the process of executing the shooting processing flow, before the target algorithm needs to be run, the system load information is obtained, and the estimated time consumption of the target algorithm is determined according to the system load information. According to the shooting timeout threshold and the estimated time consumption of the target algorithm, it is determined whether to run the target algorithm or skip the target algorithm, and the target algorithm is any one of the multiple algorithms.
[0199] In the embodiment of the present application, before the electronic device runs each algorithm in the shooting process, it can determine the estimated time consumption of the algorithm according to the system load information, and determine whether to run the algorithm or skip the algorithm according to the estimated time consumption and the shooting timeout threshold. In this way, the total running time of the shooting process can be controlled with the shooting timeout threshold as a constraint, so that the total running time of the shooting process is more likely to not exceed the shooting timeout threshold, thereby reducing the possibility of timeout of the shooting process, reducing the image loss rate during image shooting, and improving the user's shooting experience.
[0200] It should be noted that: when shooting, the shooting device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0201] The functional units and modules in the above embodiments may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit, and the above integrated units may be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.
[0202] The shooting device and shooting method embodiments provided in the above embodiments belong to the same concept. The specific working process of the units and modules in the above embodiments and the technical effects brought about can be found in the method embodiment part and will not be repeated here.
[0203] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0204] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0205] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.
[0206] The embodiments of the present application also provide a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments.
[0207] The embodiment of the present application also provides a chip system, which includes a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement the steps of any method embodiment of the present application. The chip system can be a single chip or a chip module composed of multiple chips.
[0208] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instruction may be transmitted from a website site, computer, server or data center by wired (such as: coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL), etc.) or wireless (such as: infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape, etc.), an optical medium (such as a digital versatile disc (DVD), etc.), or a semiconductor medium (such as a solid state disk (SSD), etc.).
[0209] The above are optional embodiments provided for the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the technical scope disclosed in the present application shall be included in the protection scope of the present application.
Claims
1. A shooting method, characterized in that: The method comprises: Receive shooting instructions; In response to the shooting instruction, executing a shooting process to obtain a target image, the shooting process includes a plurality of algorithms executed in sequence, and the plurality of algorithms are used to process the original image data collected by the camera to obtain the target image; In the process of executing the shooting process, before the target algorithm needs to be run, system load information is obtained, the target algorithm is any one of the multiple algorithms, and the system load information includes the running status information of each chip in the multiple chips; the load deterioration coefficient of each chip in the multiple chips is determined according to the running status information of each chip in the multiple chips, and the load deterioration coefficient of the chip is used to indicate the degree of deterioration of the current running status of the chip compared with the normal running status of the chip; according to the weight of each chip in the multiple chips, the load deterioration coefficient of each chip in the multiple chips is weighted averaged to obtain a target deterioration coefficient, and the weight of the chip is used to indicate the importance of the chip when the target algorithm is running; the target deterioration coefficient is multiplied by the preset time consumption corresponding to the target algorithm to obtain the estimated time consumption of the target algorithm; the target duration is obtained, and the target duration is obtained by subtracting the current running duration of the shooting process from the shooting timeout threshold; if the estimated time consumption of the target algorithm is less than or equal to the target duration, the target algorithm is run; if the estimated time consumption of the target algorithm is greater than the target duration, the target algorithm is skipped.
2. The method according to claim 1, characterized in that The plurality of chips are chips required to execute the shooting process.
3. The method according to claim 2, characterized in that The multiple chips include a chip where a central processing unit (CPU) is located, a chip where a graphics processing unit (GPU) is located, and a chip where a digital signal processor (DSP) is located.
4. The method according to claim 1, characterized in that The operating status information of the chip includes the load rate and frequency of the chip.
5. The method according to any one of claims 1 to 4, characterized in that: Determining the load deterioration coefficient of each chip according to the operating status information of each chip in the plurality of chips comprises: For the operation status information of any one chip among the plurality of chips, the operation status information of the chip is input into an operation performance model corresponding to the chip to obtain a load deterioration coefficient output by the operation performance model.
6. An electronic device, characterized in that: The electronic device includes: one or more processors, and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the method as claimed in any one of claims 1 to 5.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises instructions, and when the instructions are executed on an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 5.
8. A computer program product, characterized in that When the computer program product is executed on an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 5.
