Image shooting method, device, chip, terminal and storage medium

By performing preliminary frame selection processing in the front-end chip and transmitting candidate image frames to the back-end chip, the problem of high data transmission pressure in zero-delay photography is solved, and image shooting efficiency and speed are improved.

CN115225819BActive Publication Date: 2024-11-19伟光有限公司(CN)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210863143.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-07-20
Publication Date
2024-11-19
Estimated Expiration
2042-07-20

AI Technical Summary

Technical Problem

During the zero-delay photography process, the terminal needs to transmit a large amount of image data to the back-end chip, resulting in high data transmission pressure, affecting the photo shooting rate and image processing speed.

Method used

The preliminary frame selection process is performed in the front-end chip, the candidate image frame is selected, and transmitted to the back-end chip, and the back-end chip then performs the final frame selection process, thereby reducing the data transmission pressure between the front-end chip and the back-end chip.

Benefits of technology

By sharing the frame selection task, the data transmission pressure of the terminal and the processing pressure of the back-end chip are reduced, and the image shooting efficiency and data transmission speed are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115225819B_ABST
    Figure CN115225819B_ABST
Patent Text Reader

Abstract

The present application discloses an image shooting method, device, chip, terminal and storage medium, and relates to the field of image shooting technology. The method includes: the back-end chip sends a shooting instruction to the front-end chip, the front-end chip responds to the shooting instruction sent by the back-end chip, reads the cached image frame from the memory, the cached image frame is output by the image sensor and cached in the memory, based on the target frame selection strategy, the front-end chip selects a candidate image frame from the cached image frame, and transmits the candidate image frame to the back-end chip, the back-end chip receives the candidate image frame transmitted by the front-end chip, the candidate image frame is selected from the cached image frame by the front-end chip based on the target frame selection strategy, and the number of candidate image frames is less than the number of cached image frames, and the back-end chip selects the target image frame for generating the photographed image from the candidate image. The embodiment scheme of the present application reduces the pressure on the terminal to transmit image data in zero-delay photography.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image capture technology, and in particular to an image capture method, device, chip, terminal and storage medium. Background Art

[0002] Different from traditional photo taking methods, during zero-delay photo taking, the terminal caches part of the preview image data while previewing the image. Then, based on the shooting instruction, the terminal selects the final display image from the cached image data, realizing “what you shoot is what you get”.

[0003] In related technologies, the process of selecting cached images is usually performed in the back-end chip. In order to obtain higher-quality captured images, the terminal often needs to transmit more cached full-frame image data backward, which in turn causes greater transmission pressure on the terminal and affects the shooting rate. Summary of the invention

[0004] The embodiments of the present application provide an image shooting method, device, chip, terminal and storage medium, which can reduce the data transmission pressure of the terminal in the zero-delay shooting process. The technical solution is as follows:

[0005] On the one hand, an embodiment of the present application provides an image capturing method, which is used in a front-end chip, and the method includes:

[0006] In response to a shooting instruction sent by a back-end chip, a cached image frame is read from a memory, where the cached image frame is output by an image sensor and cached in the memory;

[0007] Based on the target frame selection strategy, selecting candidate image frames from the cached image frames, the number of frames of the candidate image frames being smaller than the number of frames of the cached image frames;

[0008] The candidate image frames are transmitted to the back-end chip so that the back-end chip selects a target image frame for generating a photographed image from the candidate image frames.

[0009] On the other hand, an embodiment of the present application provides an image capturing method, which is used for a back-end chip, and includes:

[0010] Sending a shooting instruction to the front-end chip so that the front-end chip reads a cached image frame from the memory, wherein the cached image frame is output by the image sensor and cached in the memory;

[0011] Receiving candidate image frames transmitted by the front-end chip, the candidate image frames are selected by the front-end chip from the cached image frames based on a target frame selection strategy, and the number of frames of the candidate image frames is less than the number of frames of the cached image frames;

[0012] A target image frame for generating a photographic image is selected from the candidate images.

[0013] On the other hand, an embodiment of the present application provides an image capturing device, the device comprising:

[0014] A reading module, configured to read a cached image frame from a memory in response to a shooting instruction sent by a back-end chip, wherein the cached image frame is output by an image sensor and cached in the memory;

[0015] A first frame selection module, configured to select candidate image frames from the cached image frames based on a target frame selection strategy, wherein the number of frames of the candidate image frames is smaller than the number of frames of the cached image frames;

[0016] The transmission module is used to transmit the candidate image frames to the back-end chip so that the back-end chip selects a target image frame for generating a photographed image from the candidate image frames.

[0017] On the other hand, an embodiment of the present application provides an image capturing device, the device comprising:

[0018] A sending module, used for sending a shooting instruction to the front-end chip, so that the front-end chip reads a cached image frame from the memory, wherein the cached image frame is output by the image sensor and cached in the memory;

[0019] A receiving module, configured to receive candidate image frames transmitted by the front-end chip, wherein the candidate image frames are selected by the front-end chip from the cached image frames based on a target frame selection strategy, and the number of frames of the candidate image frames is less than the number of frames of the cached image frames;

[0020] The second frame selection module is used to select a target image frame for generating a photographed image from the candidate images.

[0021] On the other hand, an embodiment of the present application provides a chip, which includes a programmable logic circuit and / or program instructions. When the chip is running, it is used to implement the image capture method of the front-end chip or the back-end chip side as described in the above aspects.

[0022] On the other hand, an embodiment of the present application provides a terminal, which includes a front-end chip, a back-end chip and a memory, and the at least one instruction is loaded and executed by the front-end chip and the back-end chip to implement the image capture method as described in the above aspects.

[0023] On the other hand, an embodiment of the present application provides a computer-readable storage medium, in which at least one program is stored, and the at least one instruction is loaded and executed by a processor to implement the image capture method as described in the above aspects.

[0024] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the image capture method described in the above aspect.

[0025] In an embodiment of the present application, when a photo-taking instruction is received, the front-end chip reads the cached image frames from the memory, and first performs frame selection processing on the cached image frames based on the target frame selection strategy, and then transmits the obtained candidate image frames to the back-end chip. The back-end chip determines an appropriate frame selection strategy based on the target frame selection strategy transmitted by the front-end chip and performs final frame selection processing on the candidate image frames, and finally obtains the target image frame for generating the photographed image; through the scheme of the present application, the front-end chip first performs preliminary frame selection, which reduces the pressure on transmitting data to the back-end chip, improves the data transmission speed between the front-end chip and the back-end chip in the image shooting scenario, and reduces the subsequent frame selection pressure of the back-end chip, which helps to improve image shooting efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 A structural block diagram of a terminal provided by an exemplary embodiment of the present application is shown;

[0028] Figure 2 A flowchart of an image capturing method provided by an exemplary embodiment of the present application is shown;

[0029] Figure 3 A schematic diagram showing a principle of an image capturing process provided by an exemplary embodiment of the present application is shown;

[0030] Figure 4 A flowchart of a frame selection process provided by an exemplary embodiment of the present application is shown;

[0031] Figure 5 A flowchart of a process for adjusting the number of memory cache frames provided by an exemplary embodiment of the present application is shown;

[0032] Figure 6 A flowchart of an image capturing method provided by another exemplary embodiment of the present application is shown;

[0033] Figure 7A schematic diagram showing a flow chart of an image capturing system provided by an exemplary embodiment of the present application is shown;

[0034] Figure 8 A structural block diagram of an image capturing device provided by an exemplary embodiment of the present application is shown;

[0035] Fig. 9 A structural block diagram of an image capturing device provided by another exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0036] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0037] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0038] For ease of understanding, the nouns involved in the embodiments of the present application are explained below.

[0039] Zero Shutter Lag (ZSL): A photography solution that reduces the delay in taking photos. In the normal photography process, after the shooting command is issued, the camera will perform a series of adjustments and corrections, such as focus, exposure, white balance, etc., and then encode and process the image obtained by the camera when the correction is completed. Not only is the final displayed image not the framing screen when the shooting command is issued, but the shooting time is long and there is a more obvious delay. Compared with ordinary photography, zero-delay photography stores the preview image as cached image data in the buffer. When a shooting command is issued, the terminal determines the actual shutter time, and selects the appropriate image data from the cached image through frame selection processing as the final display image, and then transmits the image to the image post-processing algorithm module, which saves the camera calibration time after the shooting command is issued, and realizes "what you shoot is what you get".

[0040] Frame selection is a common processing method during mobile phone shooting. Compared with the traditional shooting process, the terminal first adjusts the shooting parameters in response to the shooting command, and obtains the image data after the shooting parameters are configured. In the zero-delay shooting process, in order to present a better image, the terminal first stores part of the image data for preview as an alternative image. When receiving the shooting command, the terminal selects a more suitable image from the alternative images as the shooting image, where the standard for selecting a suitable image can be image clarity or image acquisition time. In the related technology, since the frame selection process is completed in the back-end chip, the terminal needs to transmit the stored image data to the back-end chip, and the transmitted data is often a whole frame of the image. This transmission process puts a lot of pressure on the data transmission of the terminal, which in turn affects the image processing speed and shooting effect.

[0041] In order to reduce the pressure of the terminal to transmit image data backward during zero-delay photography, an embodiment of the present application provides a zero-delay photography solution in which a front-end chip and a back-end chip cooperate in frame selection. Using this solution, when the terminal receives a photo request, the front-end chip first performs preliminary frame selection processing on the cached image frames read from the memory. Through the preliminary frame selection processing, the front-end chip selects candidate image frames from the cached image frames based on the target frame selection strategy, and then the front-end chip only needs to transmit the candidate image frames to the back-end chip, thereby reducing the pressure of the front-end chip to transmit image data to the back-end chip and improving the transmission speed; and the back-end chip only needs to perform further frame selection processing on the candidate image frames, reducing the processing pressure on the back-end chip.

[0042] Please refer to Figure 1 , which shows a block diagram of a terminal provided by an exemplary embodiment of the present application. The terminal 100 may include one or more of the following components: a front-end chip 110 , a back-end chip 120 , a memory 130 , and an image sensor 140 .

[0043] The front-end chip 110 is used to pre-process the image data and send the pre-processed data to the back-end chip 120. In the embodiment of the present application, an ISP (Image Signal Processing) is provided in the front-end chip 110, which is used to perform scene recognition analysis and preliminary selection on the image data output by the image sensor 140, and transmit the scene recognition results and the selected data to the back-end chip 120, which helps to reduce the pressure of the front-end chip 110 to transmit image data to the back-end chip 120. Of course, in other possible embodiments, other types of processors can also be provided in the front-end chip 110, and this embodiment is not limited to this. It should be noted that compared with the back-end chip 120, the front-end chip 110 has image recognition capabilities and can perform scene recognition analysis on the obtained image.

[0044] The back-end chip 120 is used to perform the final frame selection processing on the candidate images and perform image post-processing on the selected images. Optionally, the back-end chip 120 can be implemented as an application processor (AP). In the embodiment of the present application, the back-end chip may include one or more processing cores. The back-end chip 120 uses various interfaces and lines to connect the various parts of the entire terminal 100, and executes various functions and processes data of the terminal 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 130, and calling data stored in the memory 130. Optionally, the back-end chip 120 can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The back-end chip 120 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), and a modem. The CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content that needs to be displayed on the touch screen; the NPU is used to implement artificial intelligence (AI) functions; and the modem is used to process wireless communications. It is understandable that the above-mentioned modem may not be integrated into the back-end chip 120, but may be implemented by a separate chip.

[0045] The memory 130 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 130 includes a non-transitory computer-readable storage medium. The memory 130 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 130 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc.; the data storage area may store data created according to the use of the terminal 100 (such as audio data, a phone book), etc.

[0046] The image sensor 140 is used to convert the image optical signal into an electronic signal and transmit the image data backward. In the embodiment of the present application, the memory 130 stores the image data transmitted by the image sensor 140, and the image data is used for the front-end chip 110 to perform scene recognition, and is also used for the front-end chip 110 and the back-end chip 120 to select a suitable image as a photographic image.

[0047] In addition, those skilled in the art will appreciate that the structure of the terminal 100 shown in the above drawings does not constitute a limitation on the terminal, and the terminal may include more or fewer components than shown, or combine certain components, or arrange the components differently. For example, the terminal 100 also includes a display screen, a speaker, an input unit, a sensor (such as an acceleration sensor, an angular velocity sensor, a light sensor, etc.), an audio circuit, a WiFi module, a power supply, a Bluetooth module, and other components, which will not be described in detail here.

[0048] Please refer to Figure 2 , which shows a flowchart of an image capturing method provided by an exemplary embodiment of the present application. This embodiment uses the method for Figure 1 Taking the terminal shown as an example, the method may include the following steps.

[0049] Step 201: The back-end chip sends a shooting instruction to the front-end chip so that the front-end chip reads a cached image frame from the memory. The cached image frame is output by the image sensor and cached in the memory.

[0050] In the zero-delay photography process, when the photography application is in the preview state, the terminal adjusts the control parameters of the camera based on the image data for preview, and then the terminal caches part of the image data transmitted by the image sensor into the memory, so that the terminal can quickly select the photography image from the cached image. When the user performs a photography operation in the photography application, the back-end chip responds to the photography request and first sends a photography instruction to the front-end chip, wherein the photography instruction is used to instruct the front-end chip to read the cached image frame in the memory.

[0051] Step 202, in response to the shooting instruction sent by the back-end chip, the front-end chip reads the cached image frame from the memory, and the cached image frame is output by the image sensor and cached in the memory.

[0052] When the front-end chip receives the photo command sent by the back-end chip, in response to the photo command, the front-end chip reads the cached image frame cached in the memory. The cached image frame read by the front-end chip is the spare image frame for the photo operation, and the terminal selects a suitable photo image from the cached image frame through frame selection processing.

[0053] Step 203: Based on the target frame selection strategy, the front-end chip selects candidate image frames from the cached image frames, and the number of candidate image frames is smaller than the number of cached image frames.

[0054] When the front-end chip reads the cached image frame, it first selects the cached image frame based on the target frame selection strategy. Compared with the related art, in response to the photo request of the photo application, the terminal directly transmits the image data cached in the form of a whole frame to the back-end chip, and the pressure on the terminal to transmit image data is relatively large. The front-end chip reduces the image data that needs to be transmitted backward by making a preliminary selection of the cached image frame, thereby reducing the pressure on the terminal to transmit image data.

[0055] In a possible implementation, when the front-end chip determines the target frame selection strategy, the front-end chip selects the cached image frames, and selects the image frames that meet the target frame selection strategy as candidate image frames. The target frame selection strategy is a strategy that includes image selection criteria, and the front-end chip determines the image selection criteria that meet the actual needs of the user based on the current shooting scene and control parameters. The target frame selection strategy includes multiple target frame selection sub-strategies, and the target frame selection sub-strategies have different strategy priorities. When selecting frames based on the target frame selection strategy, the front-end chip selects the cached image frames in turn according to the strategy priorities of the target frame selection sub-strategies, and then selects the candidate image frames that meet the target frame selection strategy.

[0056] Optionally, the target frame selection strategy can be a fixed frame selection strategy determined by the front-end chip. For example, in zero-delay photography, the front-end chip uses a frame selection strategy that prioritizes clarity and secondly time as a fixed target frame selection strategy, that is, first selects image frames that meet the target clarity from the cached image frames, and then selects image frames whose timestamps are close to the time when the shooting request is issued from the above image frames, to obtain candidate image frames.

[0057] In different shooting scenarios, users have different requirements for photographic images, and the criteria for judging the quality of image data are not fixed. In one possible implementation, the front-end chip can dynamically determine the target frame selection strategy, that is, the front-end chip can dynamically determine the frame selection strategy that meets the current photography requirements based on the control parameters and scene information when the shooting request is issued, that is, dynamically determine the target frame selection sub-strategy contained in the target frame selection strategy, and dynamically determine the strategy priority of each target frame selection sub-strategy. Among them, the control parameters may include the shooting mode adopted by the camera application, such as portrait mode, still life mode, night scene mode, super night mode, etc. The control parameters may also include the setting parameters of the terminal camera, such as white balance, automatic exposure, etc., that is, the control parameters can reflect the user's real-time shooting needs.

[0058] Step 204 : The front-end chip transmits the candidate image frames to the back-end chip so that the back-end chip selects a target image frame for generating a photographed image from the candidate image frames.

[0059] Based on step 203, the front-end chip only transmits backward the image frames in the cached image frames that meet the target frame selection strategy, so the pressure of the front-end chip to transmit the image frames to the back-end chip in this step is relatively small.

[0060] In zero-delay photography, the back-end chip also has the ability to select frames, and can select candidate image frames according to a certain frame selection strategy. In one possible implementation, based on the target selection strategy, the front-end chip selects at least two candidate image frames from the cached image frames, and then the front-end chip transmits the candidate image frames to the back-end chip through the MIPI (Mobile Industry Processor Interface) interface, and the back-end chip further selects the image frames.

[0061] Step 205 , the back-end chip receives the candidate image frames transmitted by the front-end chip, the candidate image frames are selected by the front-end chip from the cached image frames based on the target frame selection strategy, and the number of candidate image frames is less than the number of cached image frames.

[0062] Based on the front-end chip transmitting the candidate image frames backward, the back-end chip receives the candidate image frames. In the embodiment of the present application, in order to reduce the image data transmission pressure of the terminal, the terminal first preliminarily selects the cached image frames through the front-end chip. At the same time, the front-end chip still transmits a certain number of candidate image frames to the back-end chip, that is, the terminal still retains the processing flow of the back-end chip selecting the image frames. Therefore, when the terminal does not have the corresponding front-end chip, frame selection can still be achieved, and the coupling risk of the front-end chip and the back-end chip is reduced.

[0063] Step 206: The back-end chip selects a target image frame for generating a photographed image from the candidate image frames.

[0064] In the zero-delay photography process, in response to a photography request, the terminal needs to select a target image frame suitable for the photographed image from all cached image frames. In a possible implementation, the candidate image frames have at least two image frames, and the back-end chip selects the candidate image frames according to a certain frame selection strategy to obtain a target image frame.

[0065] It should be noted that after obtaining the target image frame, the back-end chip performs image post-processing on the target image frame to obtain the photographed image. The image post-processing may include adding tag information to the image frame, such as adding image width and height information, exposure parameters, etc., and may also include adjusting the image size to obtain an image frame thumbnail, and further performing JPEG encoding processing on the original image frame and the thumbnail to obtain image data in JPEG format.

[0066] To sum up, in the embodiment of the present application, when performing zero-delay photography, the front-end chip responds to the photography instruction sent by the back-end chip, reads the cached image frames in the memory, and selects the image frames that meet the target frame selection strategy, and then transmits them to the back-end chip as candidate image frames, and the back-end chip selects the target image frames for generating the final photographed image; before the front-end chip transmits image data to the back-end chip, the front-end chip first performs a preliminary selection of the cached image frames to obtain candidate image frames, and the number of candidate image frames is less than the cached image frames. Compared with the related art, when the back-end chip undertakes the frame selection task, the terminal transmits all the image data cached in a whole frame manner to the back-end chip. The embodiment of the present application divides the frame selection task into two parts, which are completed jointly by the front-end chip and the back-end chip. While reducing the pressure on the terminal to transmit image data backward, it also reduces the processing pressure on the back-end chip, thereby improving the processing rate of the frame selection process.

[0067] During the zero-delay photography process, the memory caches part of the image data before the photography command is issued, thereby providing different alternative possibilities for the photographed image. In different photography scenarios, users often have different requirements for photographed images. In related technologies, the back-end chip is limited to only being able to complete frame selection based on fixed parameters, that is, it can only select photographed images from cached image frames based on fixed frame selection strategies. In an embodiment of the present application, the front-end chip can perform scene recognition on the cached image and determine the real-time scene information, and then the front-end chip can determine the frame selection strategy that meets the shooting requirements based on the shooting scene. In one possible implementation, the front-end chip dynamically determines the target frame selection strategy to achieve frame selection, which may include the following steps.

[0068] 1. The front-end chip performs scene recognition on the cached image frames to obtain scene recognition results.

[0069] Since the front-end chip is integrated with ISP, the front-end chip has the ability to perform scene recognition on image data. Through scene recognition, the front-end chip can judge the current shooting scene from multiple image frames, which is the scene recognition result. For example, when the front-end chip obtains multiple image frames with scenery as the content through scene recognition, the front-end chip obtains the scene recognition result that the current shooting scene is scenery.

[0070] In a possible implementation, Figure 3 As shown, the front-end chip 310 reads the cached image frame 331 from the memory 330. After the cached image frame 331 is transmitted to the front-end chip, the front-end chip performs scene recognition on the read cached image frame 331 through the scene recognition unit 311. By comparing the contents in multiple cached image frames, the front-end chip determines the subject in the current shooting scene, that is, obtains the scene recognition result 312. For example, when the portrait is in the center of the image, or occupies a large proportion in the overall image, the front-end chip determines that the current scene is a portrait scene; or, when the brightness of multiple cached image frames is low, the front-end chip determines that the current scene is a night scene.

[0071] Optionally, the front-end chip can analyze and identify the current shooting scene in combination with the control parameters used in the photo application, wherein the control parameter can be the image mode used by the photo application, and the front-end chip can judge the current shooting scene by the image mode. In the related art, the photo application often provides users with a variety of photo modes, such as portrait mode, landscape mode, night scene mode, super night scene mode, etc., so that users can quickly adjust the terminal settings based on photo needs. For example, when the user takes a portrait as the shooting content, the user can choose to apply the portrait mode, and then the terminal adjusts the image shooting module such as the camera to adapt to the parameters of the portrait shooting. The control parameter can also be 3A statistical information (auto focus, auto exposure, automatic white balance). For example, the front-end chip can judge the brightness of the current scene by the AE (Auto Exposure) data in the 3A statistical information, and then the front-end chip can determine that the current scene is a night scene mode or a super night scene mode. It should be noted that the above control parameters and corresponding scene judgments are for illustration only. The control parameters can also include other multiple parameters set by the terminal during the image shooting process. Based on a control parameter, the front-end chip can judge the scene from multiple angles, and the embodiment of the present application is not limited to this.

[0072] Optionally, during the process of scene recognition, the front-end chip can set feature points in the cached image frames according to the image content, and then analyze the feature point change characteristics, that is, analyze the motion characteristics of the subject, and the front-end chip determines the scene analysis results in combination with the feature point change characteristics and the scene content. For example, based on the small changes in feature points in multiple image frames, the front-end chip can determine that the current shooting scene is a static scene; and when the positions of feature points in multiple image frames have large changes, or the positional relationship between multiple feature points has large changes, the front-end chip can determine that the current shooting scene is a dynamic scene based on the feature point change characteristics, and can determine the change rate characteristics of the dynamic scene based on the degree of change in the position of the feature points in the previous and next image frames, such as whether the current scene is a high-speed scene, a snapshot scene, etc. The above scene recognition results determined based on the feature point change characteristics are for illustrative purposes only, and this application does not limit this.

[0073] 2. Determine the target frame selection strategy based on the scene recognition result. The target frame selection strategy is used to indicate the selection of image frames that meet the shooting requirements of the current shooting scene, and different scenes correspond to different frame selection strategies.

[0074] After obtaining the scene recognition result, the front-end chip can determine the current shooting scene. Corresponding to different shooting scenes, users have different evaluation criteria for images, that is, different requirements for photographed images. For example, in a snapshot scene, users hope that "what you shoot is what you get". Therefore, the image frames in the cached image frames that are acquired close to the time when the user takes the photo should be the better image frames. Figure 3 As shown, the front-end chip 310 determines the user's demand for the photographed image based on the scene recognition result, and then determines which image frame meets the current shooting scene requirement, that is, determines the target frame selection strategy 313, so that the first frame selection unit 314 performs frame selection processing on the cached image frame 331 based on the target frame selection strategy 313.

[0075] Based on the scene recognition result, the front-end chip determines the target frame selection sub-strategy and the policy execution priority corresponding to the target frame selection sub-strategy from the candidate frame selection sub-strategies, and determines the target frame selection sub-strategy and the policy execution priority as the target frame selection strategy.

[0076] Based on the different shooting scenes represented by the scene recognition results, the front-end chip determines different target frame selection strategies so as to select frames according to the frame selection criteria suitable for the current shooting scene. Among them, the target frame selection strategy is composed of multiple target frame selection sub-strategies, each target frame selection sub-strategy corresponds to a frame selection standard, and based on the scene recognition results, different target frame selection sub-strategies have different strategy execution priorities. When the front-end chip performs frame selection, it preferentially executes the target frame selection sub-strategy with a high strategy execution priority. Correspondingly, the front-end chip can choose which frame selection sub-strategy to call from multiple candidate frame selection sub-strategies to adapt to different shooting scenes to form a target frame selection strategy that meets real-time scenes and user needs. Candidate frame selection sub-strategies may include time strategies, clarity strategies, ratio strategies, brightness strategies, etc., which are not limited in the embodiments of the present application.

[0077] When the scene recognition result is a static shooting scene, the target frame selection sub-strategy includes at least a timestamp-based frame selection sub-strategy and an image quality-based frame selection sub-strategy, and the policy execution priority of the image quality-based frame selection sub-strategy is higher than the policy execution priority of the timestamp-based frame selection sub-strategy.

[0078] When the front-end chip determines that the current scene is a static shooting scene, the front-end chip can determine that the image frame with higher image quality in the current scene is a better image frame. Therefore, the front-end chip should determine that the target frame selection strategy must include a frame selection sub-strategy with image quality as the frame selection standard; in the zero-delay shooting process, the terminal calculates the time when the user performs the shooting operation, and selects the preview image that the terminal is displaying when the user presses the shutter as the shooting image as much as possible, so as to achieve zero delay in image presentation, that is, "what you shoot is what you get". Therefore, the target frame selection sub-strategy should include a frame selection sub-strategy based on timestamps. Based on this frame selection sub-strategy, the front-end chip selects images whose acquisition time is close to the shutter time. Based on the fact that in static shooting, the content between the previous and next image frames does not change much, and thus the difference between the frames of images with different acquisition times is not much, the front-end chip should first ensure the image quality, and first select the images that meet the image quality standard, that is, the frame selection sub-strategy based on image quality should have a higher strategy execution priority. It should be noted that the frame selection sub-strategy based on image quality can be a clarity strategy, an exposure strategy, etc., which is not limited in the embodiments of the present application.

[0079] In one possible implementation, the front-end chip performs scene recognition on multiple cached image frames, and based on the fact that the feature points in the image frames change little between the previous and next frames, and the camera application uses still life mode or portrait mode for shooting, the front-end chip determines that the current shooting scene is a static scene. The front-end chip calls the frame selection sub-strategy based on image clarity and the frame selection sub-strategy based on timestamp from the candidate frame selection sub-strategies, and the strategy execution priority of the clarity strategy is higher than the time strategy. Among them, the frame selection sub-strategy based on image clarity means that the front-end chip selects some image frames with better clarity from multiple image frames, and the frame selection sub-strategy based on timestamp means that the front-end chip selects some image frames with timestamps close to the shutter time from multiple image frames, and based on the strategy execution priority, the front-end chip first selects image frames with better clarity from the cached image frames, and then selects image frames that meet the time strategy from the above-mentioned partial image frames.

[0080] Optionally, when the scene is a still life shooting scene, the front-end chip can also combine other frame selection sub-strategies for frame selection, such as ratio strategy, exposure strategy, etc. For example, the front-end chip can also combine the frame selection sub-strategy based on the proportion of the subject to combine the target frame selection strategy. Among them, the frame selection sub-strategy based on the proportion of the subject means that the front-end chip calls the ratio parameter in the tuning file on the basis of determining the subject. The ratio parameter indicates the optimal range of the proportion of the image occupied by the subject in most photographed images. The above ratio reflects the depth information of the subject in the image. When the proportion of the subject in the image is within a preferred range, the subject has a suitable distance relationship in the image.

[0081] In an illustrative example, based on scene recognition, the front-end chip determines that the current scene is a portrait shooting scene. On the basis of adopting the frame selection sub-strategy based on image quality and the frame selection sub-strategy based on timestamp, the target frame selection strategy also includes a frame selection sub-strategy based on the proportion of the subject, and the proportion strategy has a lower strategy execution priority.

[0082] When the scene recognition result is a dynamic shooting scene, the target frame selection sub-strategy includes at least a timestamp-based frame selection sub-strategy and an image quality-based frame selection sub-strategy, and the policy execution priority of the timestamp-based frame selection sub-strategy is higher than the policy execution priority of the image quality-based frame selection sub-strategy.

[0083] When the front-end chip determines that the current scene is a dynamic shooting scene, in the cached image frames, the difference between the previous and next image frames is large, and the corresponding image frame content when the user performs the shooting operation may have a large difference from the previous and next image frames. In order to ensure that the final photographed image content is close to the corresponding preview image during the shooting operation, the target frame selection sub-strategy includes at least a frame selection sub-strategy based on timestamps, and the front-end chip should first select image frames with timestamps close to the shutter time, that is, the time strategy should have a higher strategy execution priority; in order to ensure that the final photographed image still has good image quality in dynamic scenes, the target frame selection sub-strategy also includes at least a frame selection sub-strategy based on image quality. Similarly, it should be noted that the frame selection sub-strategy based on image quality can be a clarity strategy, an exposure strategy, etc., which is not limited in the embodiments of the present application.

[0084] In an illustrative example, in a scenario where a user takes a snapshot of a portrait, during the scene recognition process, the front-end chip sets feature points for the portrait in the cached image frame, and analyzes that the positions of the feature points between multiple image frames have large changes. Combined with the selection of portrait mode and bright exposure in the photo-taking application, it can be determined that the current snapshot process is in progress, and the user wants to obtain a preview image corresponding to when the shutter button is pressed. Therefore, the front-end chip can determine that the target frame selection strategy should include a target frame selection sub-strategy based on timestamp. When the user takes a photo in portrait mode, he often wants to obtain a clearer image. That is, the front-end chip can determine that the target frame selection strategy should also include a target frame selection sub-strategy based on image quality. In comparison, in dynamic shooting scenes such as snapshot scenes, the real-time nature of the image is more important. Therefore, the time strategy should have a higher strategy execution priority.

[0085] It should be noted that the number of target frame selection sub-strategies contained in the target frame selection strategy can be one or at least two, which is not limited in the embodiment of the present application.

[0086] In summary, when a cached image frame is read, the front-end chip first performs scene recognition on the image content of the cached image frame to determine the current shooting scene, so as to judge the current user's shooting needs in combination with the terminal control parameters, and obtain the scene recognition result. Then, the front-end chip selects a target frame selection sub-strategy suitable for the user's needs from the candidate frame selection sub-strategies based on the scene recognition result, and determines the corresponding strategy execution priority. Compared with the related art, in which the terminal only completes the frame selection through a fixed frame selection strategy, the front-end chip determines the target frame selection strategy based on scene recognition, realizes the dynamic determination of the frame selection strategy that meets the characteristics of the scene, better meets the user's differentiated shooting needs, and improves the image shooting quality in different scenes.

[0087] In order to reduce the pressure of image transmission, based on the target frame selection strategy obtained by the front-end chip in the above embodiment, the terminal first performs preliminary frame selection on the cached image frames through the front-end chip to obtain candidate image frames that meet the target frame selection sub-strategy, and then sends the candidate image frames backward, such as Figure 4 As shown, in a possible implementation manner, the method of executing the target frame selection sub-strategy on the cached image frames based on the descending order of the policy execution priority may include the following steps:

[0088] Step 401 , the front-end chip executes the first part of the target frame selection sub-strategy on the cached image frames based on the descending order of the policy execution priority to obtain candidate image frames.

[0089] Based on the above embodiment, the front-end chip determines the target frame selection strategy based on the scene recognition result, and the target frame selection strategy includes multiple target frame selection sub-strategies with different strategy execution priorities. Figure 3 As shown, when the front-end chip 310 determines the target frame selection strategy 313, the front-end chip 310 executes the target frame selection sub-strategies of the first part in descending order of the strategy execution priority, and preliminarily selects candidate image frames 315 from the cached image frames, wherein the number of candidate graphic frames should be less than the cached image frames, thereby reducing the number of image frames that the front-end chip needs to transmit backward and alleviating the transmission pressure.

[0090] In one possible manner, the target frame selection sub-strategy with the i-th target frame selection sub-strategy as the strategy execution priority is the i-th target frame selection sub-strategy, and the front-end chip calls the relevant parameters in the tuning file according to the i-th target frame selection sub-strategy. For example, when the i-th target frame selection sub-strategy is a frame selection sub-strategy based on the proportion of the photographed object, the front-end chip calls the proportion parameter to determine a better proportion range, and then the front-end chip determines the image priority of each cached image frame according to the frame selection standard in the i-th target frame selection sub-strategy, and selects the candidate image frame to be transmitted to the back-end chip based on the image priority.

[0091] The method of selecting the candidate image frame from the cached image frames based on the determined image priority may be any of the following:

[0092] 1. The front-end chip can set the number of selected image frames and the number of candidate image frames, and select the corresponding number of cached image frames as candidate image frames according to the image priority. For example, the front-end chip can determine that the first target frame selection sub-strategy selects n-2 frames of images from n cached image frames for the next frame selection, and then the i-th target frame selection sub-strategy selects ni×2 frames of images for the next frame selection.

[0093] In an illustrative example, in a snapshot scenario, the target frame selection strategy determined by the front-end chip includes a time strategy and a clarity strategy, and the former has a higher policy execution priority than the latter. When the front-end chip reads 8 cached image frames from the buffer, the front-end chip first determines the image priority based on the timestamp of each image frame, where the image with a capture time closer to the shutter time has a higher image priority. The front-end chip selects the first 6 frames of the cached image frames for clarity selection, and then the front-end chip determines the image priority of the above 6 frames based on the clarity of the image, and selects the first 4 frames as candidate image frames.

[0094] 2. The front-end chip can read the strategy threshold in the tuning file. The strategy threshold represents the degree to which the image meets the target frame selection sub-strategy. Then, based on the cached image frames reaching the strategy threshold of each target frame selection sub-strategy in sequence, the front-end chip determines the candidate image frame.

[0095] In an illustrative example, in a snapshot scenario, the target frame selection strategy determined by the front-end chip includes a time strategy and a ratio strategy, and the former has a higher priority than the latter. The front-end chip reads the tuning file to obtain that the strategy threshold of the time strategy is 0.5 seconds, that is, only image frames within 0.5 seconds from the shutter time can be used for the next frame selection; based on the tuning file, the front-end chip determines that the strategy threshold of the ratio strategy is 1 / 2-1 / 4, that is, only images in which the proportion of the subject in the image is between 1 / 2-1 / 4 can be used as candidate image frames.

[0096] It should be noted that, in a possible implementation, the front-end chip can select an image frame from the cached image frames as a candidate image frame, that is, the front-end chip completes the entire frame selection process by executing all target frame selection sub-strategies.

[0097] In step 402, the front-end chip transmits the scene recognition result and at least one of the target frame selection strategies to the back-end chip, so that the back-end chip selects the target image frame from the candidate image frames based on the scene recognition result and at least one of the target frame selection strategies.

[0098] Since the back-end chip does not have image analysis capabilities, such as Figure 3 As shown, the front-end chip 310 needs to transmit at least one of the scene recognition result 312 and the target frame selection strategy 313 to the back-end chip, so that the back-end chip can determine the target frame selection strategy suitable for the current shooting scene.

[0099] The front-end chip can transmit the target frame selection strategy to the back-end chip so that the back-end chip executes the target frame selection sub-strategy of the second part on the candidate image frame to obtain the target image frame, wherein the target frame selection sub-strategy of the first part is different from the target frame selection sub-strategy of the second part, and the policy execution priority of the target frame selection sub-strategy of the first part is higher than the policy execution priority of the target frame selection sub-strategy of the second part.

[0100] Optionally, the front-end chip may send the target frame selection strategy to the back-end chip after obtaining the target frame selection strategy, or may send the target frame selection strategy together with the candidate image frame to the back-end chip after the front-end chip completes preliminary frame selection processing.

[0101] Step 403: the back-end chip receives at least one of the scene recognition result and the target frame selection strategy transmitted by the front-end chip, where the scene recognition result is obtained by the front-end chip performing scene recognition on the cached image frame.

[0102] The back-end chip does not have image recognition capabilities. Figure 3 As shown, the back-end chip 320 receives the scene recognition result 312 or one of the target frame selection strategies 313 transmitted by the front-end chip 310 to determine which strategy to use for frame selection processing on the candidate image frames.

[0103] Step 404: The backend chip selects the target image frame from the candidate images based on the scene recognition result and at least one of the target frame selection strategies.

[0104] like Figure 3 As shown, the back-end chip 320 can determine the current shooting scene based on the scene recognition result 312, and then determine the target frame selection strategy that the back-end chip should execute, and the second frame selection unit 321 executes the above target frame selection strategy. Since the back-end chip relies on the set parameters when performing frame selection processing, the back-end chip can often only execute a frame selection strategy based on timestamps, or a frame selection strategy based on image clarity. When the target frame selection strategy determined by the front-end chip includes at least one of the time strategy or the clarity strategy, the back-end chip can execute part of the target frame selection sub-strategies in the target frame selection strategy. The way in which the back-end chip executes the target frame selection sub-strategy can be any of the following:

[0105] 1. When the back-end chip receives the target frame selection strategy transmitted by the front-end chip, the back-end chip can execute the target frame selection sub-strategy of the second part on the candidate image frame based on the descending order of the strategy execution priority to obtain the target image frame, wherein the target frame selection sub-strategy of the first part is different from the target frame selection sub-strategy of the second part, and the strategy execution priority of the target frame selection sub-strategy of the first part is higher than the strategy execution priority of the target frame selection sub-strategy of the second part.

[0106] When the front-end chip sends the target frame selection strategy to the back-end chip, the front-end chip can synchronize the executed target frame selection sub-strategy information. When the front-end chip and the back-end chip execute different target frame selection sub-strategies, based on different target frame selection sub-strategies corresponding to different frame selection criteria, the terminal can select cached image frames from multiple angles.

[0107] In an illustrative example, in a portrait shooting scene, the front-end chip determines a target frame selection strategy, which includes a clarity strategy, a proportion strategy and a time strategy, and the strategy execution priorities of the above target frame selection sub-strategies decrease in sequence. The front-end chip executes the clarity strategy and the proportion strategy in sequence, that is, first selects a suitable image based on the clarity and the proportion of the portrait being photographed, and obtains a candidate image frame. The candidate chip determines to execute a target frame selection sub-strategy different from the first part of the target frame selection sub-strategy based on the target frame selection strategy transmitted by the front-end chip, that is, the back-end chip selects the frame again based on the timestamp as the standard to obtain the target image frame. As a result, the terminal takes into account the image clarity, portrait proportion and real-time performance in the frame selection process.

[0108] 2. Due to the limitation that the back-end chip can only execute limited frame selection strategies, the back-end chip can also execute the target frame selection sub-strategy executed by the front-end chip.

[0109] In an illustrative example, in a snapshot shooting scenario, the front-end chip determines a target frame selection strategy, which includes a time strategy and a clarity strategy, and the strategy execution priorities of the above target frame selection sub-strategies decrease in sequence. The front-end chip executes the time strategy and the clarity strategy in sequence and obtains candidate image frames. After receiving the target frame selection strategy, based on the fact that the back-end chip can only execute the time strategy and the clarity strategy, the back-end chip can execute the time strategy again on the candidate image frames, that is, select a frame of image whose acquisition time is closest to the shutter time from the candidate image frames as the target image frame.

[0110] In a possible implementation, when the front-end chip only sends the scene recognition result to the back-end chip, that is, when the back-end chip only receives the scene recognition result, the back-end chip can determine the current shooting scene based on the scene recognition result, and then determine the target frame selection sub-strategy to be executed and the corresponding strategy execution priority.

[0111] In an illustrative example, the back-end chip receives a scene recognition result, determines that the current shooting scene is a snapshot scene, and receives 4 candidate image frames. The back-end chip can determine to first execute a timestamp-based frame selection sub-strategy for the candidate image frames, and then execute a frame selection sub-strategy based on image clarity. When selecting frames, the back-end chip arranges the candidate image frames in descending order according to the difference between the acquisition time and the shutter time to determine the image priority, and selects the first two frames of images therefrom, and then selects the frame with the best image clarity from the above two frames of images as the target image frame.

[0112] It should be noted that, in a possible implementation, the front-end chip can select an image frame from the cached image frames through preliminary frame selection, and transmit it to the back-end chip as a candidate image frame. At this time, the back-end chip does not need to perform frame selection processing, that is, if the processing power permits, the front-end chip can take on all frame selection tasks.

[0113] In zero-delay photography, the terminal first stores a certain number of image frames, and often stores the data of the entire frame, which will occupy a lot of system resources. Figure 5 , which actually shows a flowchart of an image capturing method provided by an exemplary embodiment of the present application. Figure 5 Taking the front-end chip of the terminal shown in FIG. 1 as an example, the method may include the following steps:

[0114] Step 501: The front-end chip reads m cached image frames from the memory in response to a shooting instruction sent by the back-end chip.

[0115] In zero-delay photography, the terminal needs to cache image data as an alternative image. The memory is often cached in whole frames, which consumes more storage space. The terminal can save storage space by determining a smaller number of cache frames. When the camera application is in preview mode, that is, when the front-end chip has not received the camera command, the memory is controlled to store the cached image frames with a smaller number of frames. When the front-end chip receives the shooting command, the front-end chip first reads the cached image frames with a smaller number of frames from the storage.

[0116] In a possible implementation, Figure 3 As shown, based on the buffer size determined by the terminal, the memory 330 caches the m frames of the original image data 340 transmitted backward by the image sensor, and updates the cached image in real time, that is, the memory 330 caches the m frames of the image data transmitted backward by the image sensor the latest. Among them, m frames are a small number of frames. For example, the terminal determines that in the preview state, the memory caches the latest 3 frames of the image in real time. When the front-end chip receives the shooting instruction transmitted by the back-end chip, in response to the shooting instruction, the front-end chip first reads the above m frames of the image from the memory.

[0117] Step 502: The front-end chip determines a target frame selection strategy based on m cached image frames.

[0118] Based on step 501, after the front-end chip reads the m-frame cache image frames in the memory, the front-end chip performs scene recognition on the m-frame images and determines the target frame selection strategy based on the scene recognition results.

[0119] Step 503, in response to the shooting instruction sent by the back-end chip, the front-end chip sends a cache increase instruction to the memory so that the memory increases the capacity of the buffer from m frames to n frames.

[0120] To save storage space of the terminal, the memory first stores cached image frames with a smaller frame number m. During the frame selection process, the cached image frames are the alternative images selected by the terminal for taking photos. To provide more frame selection possibilities, when a photo-taking instruction is received, the front-end chip sends an instruction to increase the number of cached frames to the memory, and the cached images are increased from m to n. This reduces the occupied cache space while ensuring that the frame selection process has a sufficient number of alternative images.

[0121] It should be noted that the memory updates and stores the latest image data in real time during the process of storing cached image frames. Therefore, in response to the cache increase instruction sent by the front-end chip, the memory continues to store the latest image data on the basis of retaining the previous m frames of images. That is, when the memory completes executing the cache increase instruction, the n frames of images in the memory include the m frames of cached images in step 502.

[0122] Step 504: the front-end chip reads n cached image frames from the memory.

[0123] As can be seen from step 503, in order to ensure that there are enough candidate images for the frame selection process, the front-end chip sends a cache increase instruction to make the memory store n cache image frames, where n image frames are relatively more image frames, and the n image frames are candidate image frames for the frame selection process. Thus, the front-end chip reads the above n cache image frames from the memory so that the terminal can select the final target image frame from them and generate a photographed image.

[0124] Step 505: Based on the target frame selection strategy, the front-end chip selects a candidate image frame from the n cached image frames.

[0125] Since the n-frame cache image frames include the m-frame cache image frames in step 502 , the target frame selection strategy determined based on the scene recognition result of the m-frame cache image frames can also be applied to the n-frame cache image frames.

[0126] To summarize, when the camera application is in preview state, the memory only updates a small number of cached image frames in real time, thereby saving the storage space occupied by the zero-delay camera process. When the front-end chip receives a camera command, the front-end chip only needs to read the above-mentioned small number of cached image frames, and perform scene recognition on them to determine the target frame selection strategy. When the front-end chip receives a camera command, the front-end chip sends a cache increase instruction to the memory to increase the number of cached frames, thereby ensuring the number of frames of alternative images in the frame selection process. While reducing the space required to store cached images, the image quality of the photographed images is guaranteed.

[0127] Please refer to Figure 6 , which shows a flowchart of an image capturing method provided by another exemplary embodiment. This embodiment uses the method for Figure 1 Taking the front-end chip and the back-end chip of the terminal as an example, the method may include the following steps:

[0128] Step 601: The back-end chip sends a shooting instruction to the front-end chip so that the front-end chip reads a cached image frame from a memory. The cached image frame is output by an image sensor and cached in the memory.

[0129] When a user uses the selfie application to take a photo, the back-end chip first sends a photo instruction to the front-end chip in response to the operation, so that the front-end chip can take part of the frame selection task through preliminary frame selection.

[0130] Step 602: In response to the shooting instruction sent by the back-end chip, the front-end chip reads m cached image frames from the memory.

[0131] In order to save the storage space occupied by zero-delay photography, in the embodiment of the present application, the memory first updates and caches a smaller number of cached image frames in real time, that is, caches m frames of image frames. When the front-end chip receives a shooting instruction, the front-end chip first reads the above m frames of cached image frames from the memory.

[0132] Step 603, in response to the shooting instruction sent by the back-end chip, the front-end chip performs scene recognition on the m-frame cache image frames and obtains a scene recognition result.

[0133] In order to determine a frame selection strategy that meets the user's actual photography needs and meet the user's differentiated photography needs, the front-end chip performs scene recognition on the m-frame cache image frames read to determine the current shooting scene.

[0134] Optionally, the front-end chip can determine the scene recognition result in combination with the scene content and the control parameters of the terminal.

[0135] Step 604: The front-end chip determines a target frame selection strategy based on the scene recognition result.

[0136] In order to make a preliminary selection of cached image frames and reduce the pressure of data transmission and the processing pressure of the back-end chip, the front-end chip determines the target frame selection strategy that meets the current scene based on the scene recognition results. The target frame selection strategy is composed of multiple target frame selection sub-strategies selected from the candidate frame selection sub-strategies, and each target frame selection sub-strategy has a different strategy execution priority. The strategy execution priority represents the importance of different image information in the process of selecting a better image in the current scene.

[0137] Step 605 , in response to the shooting instruction sent by the back-end chip, a cache increase instruction is sent to the memory so that the memory increases the capacity of the buffer from m frames to n frames.

[0138] Since m image frames are relatively few, for example 3 frames, there may not be enough images in the m image frames to meet user needs. To ensure the image quality of the photographed images, the front-end chip sends a cache increase instruction to the memory to increase the cache image frames to n frames.

[0139] Step 606: the front-end chip reads n cached image frames from the memory.

[0140] It can be seen from step 605 that the n-frame buffered image frames are image frames used for frame selection processing, and the front-end chip reads the n-frame buffered image frames to select image frames that meet the frame selection criteria corresponding to the target frame selection strategy.

[0141] Step 607 , the front-end chip executes the first part of the target frame selection sub-strategy on the n cached image frames based on the descending order of the policy execution priority to obtain candidate image frames.

[0142] The target frame selection strategy includes a target frame selection sub-strategy, which is divided into a first part target frame selection sub-strategy and a second part target frame selection sub-strategy. The two parts of the target frame selection sub-strategy are different and are executed by the front-end chip and the back-end chip respectively. The policy execution priority of the first part target frame selection sub-strategy is higher than the policy execution priority of the second part target frame selection sub-strategy.

[0143] In a possible implementation, the front-end chip executes the target frame selection sub-strategy of the first part in descending order of the strategy execution priority, and sequentially selects candidate image frames from the n cached image frames.

[0144] Step 608: The front-end chip transmits the scene recognition result and the target frame selection strategy to the back-end chip.

[0145] Due to the lack of image recognition capabilities, the back-end chip cannot perform scene recognition on images. In order to enable the back-end chip to select appropriate image frames according to the shooting scene reflected by the image, the front-end chip can transmit the scene recognition results and target frame selection strategy to the back-end chip so that the back-end chip can analyze the scene recognition results and determine an appropriate target frame selection strategy.

[0146] Step 609 , the back-end chip receives the target frame selection strategy and the candidate image frames transmitted by the front-end chip.

[0147] This step is the same as step 402 and step 204, and will not be described in detail here.

[0148] Step 610 , the backend chip executes the target frame selection sub-strategy of the second part on the candidate image frames based on the descending order of the policy execution priority to obtain the target image frame.

[0149] The target frame selection strategy received by the back-end chip includes a first target frame selection sub-strategy and a second target frame selection sub-strategy, and the two target frame selection sub-strategies are different. The back-end chip executes the second target frame selection sub-strategy in descending order of policy execution priority.

[0150] In an illustrative example, based on the scene recognition results, the front-end chip determines that the current scene is a snapshot scene, and then determines that the target frame selection strategy consists of a time strategy, a ratio strategy, and a clarity strategy, and the strategy execution priority of each target frame selection sub-strategy decreases in sequence, wherein the front-end chip executes the time strategy and the ratio strategy in sequence, which is the first part of the target frame selection sub-strategy, and then the back-end chip executes the clarity strategy, which is the second part of the target frame selection sub-strategy, and selects a frame with the best clarity from the candidate image frames as the target image frame.

[0151] To summarize, in an embodiment of the present application, in response to a photo-taking instruction, the front-end chip reads the cached image frames in the memory, and performs scene recognition on them to determine a target frame selection strategy suitable for the current shooting scene, and then sends a cache increase instruction to the memory, and reads cached image frames with more frames. Based on the target frame selection strategy, the front-end chip performs preliminary frame selection processing on the cached image frames to obtain candidate image frames, and then based on the target frame selection strategy and candidate image frames transmitted by the front-end chip, the back-end chip performs further frame selection processing on the candidate image frames to obtain target image frames; the embodiment of the present application reduces the storage pressure of the terminal by reducing the early cached image frames, and the front-end chip performs preliminary frame selection through a target frame selection strategy that conforms to the characteristics of the scene, thereby reducing the data transmission pressure of the terminal and obtaining a photographed image that meets the actual needs of the user.

[0152] Please refer to Figure 7, which shows a schematic diagram of the system flow of an image capturing method provided by an exemplary embodiment of the present application. Figure 1 The terminal shown is used as an example for explanation.

[0153] Step 701, when the user opens the camera through the photo application, in response to the terminal adopting the zero-delay photo mode to shoot, the terminal adjusts various photo parameters and working modes of the terminal through the viewfinder image, such as adjusting 3A statistical data.

[0154] Step 702, during the zero-delay photography process, in order to provide the user with a "zero-delay" photography experience, the terminal caches the original image data output by the image sensor in real time, so that when the user performs a photography operation, the terminal can directly select an appropriate image frame from the cached image frame as the photographed image.

[0155] Step 703, when the user issues a photo-taking request, the back-end chip sends a photo-taking instruction to the front-end chip.

[0156] Step 704, in response to the above-mentioned photo-taking instruction, the front-end chip reads the cached image frame from the memory to perform scene recognition and obtain a scene recognition result.

[0157] Step 705: Based on the above scene recognition results, the front-end chip determines a target frame selection strategy that matches the current shooting scene, and synchronizes the obtained target frame selection strategy to the back-end chip.

[0158] Step 706: When the target frame selection strategy is determined, the front-end chip performs preliminary frame selection processing on the cached image frames to obtain candidate image frames.

[0159] Step 707: the front-end chip transmits the candidate image frame to the back-end chip.

[0160] In step 708, the back-end chip performs final frame selection on the candidate image frames based on the obtained target frame selection strategy, and obtains a target image frame for generating a photographed image.

[0161] Step 709: the back-end chip performs image post-processing on the target image frame to obtain a final photographed image, wherein the image post-processing may include adding tag information, adjusting the image size, and performing JPEG encoding.

[0162] Please refer to Figure 8 , which shows a structural block diagram of an image capturing device provided by an exemplary embodiment of the present application, the device is used for a front-end chip, and the device includes:

[0163] A reading module 801 is used to read a cached image frame from a memory in response to a shooting instruction sent by a back-end chip, wherein the cached image frame is output by an image sensor and cached in the memory;

[0164] A first frame selection module 802, configured to select candidate image frames from the cached image frames based on a target frame selection strategy, wherein the number of frames of the candidate image frames is smaller than the number of frames of the cached image frames;

[0165] The transmission module 803 is used to transmit the candidate image frames to the back-end chip so that the back-end chip selects a target image frame for generating a photographed image from the candidate image frames.

[0166] Optionally, the first frame selection module 802 is further configured to:

[0167] Performing scene recognition on the cached image frame to obtain a scene recognition result;

[0168] The target frame selection strategy is determined based on the scene recognition result, and the target frame selection strategy is used to indicate the selection of image frames that meet the shooting requirements corresponding to the current shooting scene, and different scenes correspond to different frame selection strategies.

[0169] Optionally, in the process of determining the target frame selection strategy based on the scene recognition result, the first frame selection module 802 is further used to:

[0170] Based on the scene recognition result, determining a target frame selection sub-strategy from candidate frame selection sub-strategies, and a strategy execution priority corresponding to the target frame selection sub-strategy;

[0171] Determine the target frame selection sub-strategy and the strategy execution priority as the target frame selection strategy;

[0172] The step of selecting a candidate image frame from the cached image frames based on the target frame selection strategy includes:

[0173] Based on the descending order of the policy execution priority, the target frame selection sub-policy is executed on the cached image frame to obtain the candidate image frame.

[0174] In the case where the scene recognition result is a static shooting scene, the target frame selection sub-strategy includes at least a frame selection sub-strategy based on timestamp and a frame selection sub-strategy based on image quality, and the policy execution priority of the frame selection sub-strategy based on image quality is higher than the policy execution priority of the frame selection sub-strategy based on timestamp;

[0175] When the scene recognition result is a dynamic shooting scene, the target frame selection sub-strategy includes at least a timestamp-based frame selection sub-strategy and an image quality-based frame selection sub-strategy, and the strategy execution priority of the timestamp-based frame selection sub-strategy is higher than the strategy execution priority of the image quality-based frame selection sub-strategy.

[0176] Optionally, in the process of executing the target frame selection sub-strategy on the cached image frame in descending order of the policy execution priority to obtain the candidate image frame, the first frame selection strategy 802 is further used to:

[0177] Based on the descending order of the policy execution priority, the target frame selection sub-policy of the first part is executed on the cached image frame to obtain the candidate image frame.

[0178] Optionally, the transmission module 803 is further used to:

[0179] The target frame selection strategy is transmitted to the back-end chip so that the back-end chip executes the target frame selection sub-strategy of the second part on the candidate image frame to obtain the target image frame, wherein the target frame selection sub-strategy of the first part is different from the target frame selection sub-strategy of the second part, and the strategy execution priority of the target frame selection sub-strategy of the first part is higher than the strategy execution priority of the target frame selection sub-strategy of the second part.

[0180] Optionally, the transmission module 803 is further used to:

[0181] The scene recognition result and at least one of the target frame selection strategies are transmitted to the back-end chip so that the back-end chip selects the target image frame from the candidate image frames based on the scene recognition result and at least one of the target frame selection strategies.

[0182] Optionally, the reading module 801 is further used for:

[0183] In response to the shooting instruction sent by the back-end chip, n frames of the cached image frames are read from the memory.

[0184] Optionally, the reading module 801 is further used for:

[0185] In response to the shooting instruction sent by the back-end chip, read m frames of the cached image frames from the memory;

[0186] In response to the shooting instruction sent by the back-end chip, a cache increase instruction is sent to the memory so that the memory increases the capacity of the buffer zone from m frames to n frames;

[0187] Read n frames of the cached image frames from the memory.

[0188] Optionally, the first frame selection module 802 is further configured to:

[0189] Determine the target frame selection strategy based on the m frames of cached image frames;

[0190] Based on the target frame selection strategy, the candidate image frame is selected from the n cached image frames.

[0191] Please refer to Fig. 9 , which shows a structural block diagram of an image capturing device provided by an exemplary embodiment of the present application, the device is used for a back-end chip, and the device includes:

[0192] The sending module 901 is used to send a shooting instruction to the front-end chip so that the front-end chip reads a cached image frame from the memory, and the cached image frame is output by the image sensor and cached in the memory;

[0193] A receiving module 902 is used to receive candidate image frames transmitted by the front-end chip, where the candidate image frames are selected by the front-end chip from the cached image frames based on a target frame selection strategy, and the number of frames of the candidate image frames is less than the number of frames of the cached image frames;

[0194] The second frame selection module 903 is used to select a target image frame for generating a photographed image from the candidate images.

[0195] Optionally, the receiving module 902 is further configured to:

[0196] Receive at least one of the scene recognition result and the target frame selection strategy transmitted by the front-end chip, wherein the scene recognition result is obtained by the front-end chip performing scene recognition on the cached image frame.

[0197] Optionally, the second frame selection module 903 is further used to:

[0198] Based on at least one of the scene recognition result and the target frame selection strategy, the target image frame is selected from the candidate images.

[0199] Optionally, when the target frame selection strategy includes a target frame selection sub-strategy and a strategy execution priority of the target frame selection sub-strategy, and the candidate image frame is obtained by the front-end chip executing the first part of the target frame selection sub-strategy on the cached image frame based on the descending order of the strategy execution priority, the receiving module 902 is further used to:

[0200] Receiving the target frame selection strategy transmitted by the front-end chip;

[0201] Optionally, the second frame selection module 903 is further used to:

[0202] Based on the descending order of the policy execution priority, the target frame selection sub-strategy of the second part is executed on the candidate image frame to obtain the target image frame, wherein the target frame selection sub-strategy of the first part is different from the target frame selection sub-strategy of the second part, and the policy execution priority of the target frame selection sub-strategy of the first part is higher than the policy execution priority of the target frame selection sub-strategy of the second part.

[0203] In summary, in the process of zero-delay photography, in response to the photography instruction sent by the back-end chip through the sending module, the reading module of the front-end chip reads the cached image frame from the memory, and the first frame selection module performs scene recognition on the cached image frame. Based on the scene recognition result, the front-end chip dynamically determines the target frame selection strategy suitable for the current shooting scene through the first frame selection module, and then, based on the obtained target selection strategy, selects the candidate image frame that meets the target frame selection strategy from the cached image frame, so that the transmission module transmits the candidate image frame to the back-end chip; when the back-end chip receives the candidate image frame through the receiving module, the second frame selection module further selects the candidate image frame, and finally obtains the target image frame for generating the photographed image. Through the embodiment of the present application, the front-end chip performs preliminary frame selection on the cached image frame. After selecting the image frame that meets the target frame selection strategy, the front-end chip only needs to transmit part of the image in the cached image frame, that is, the candidate image frame to the back-end chip, which reduces the pressure on the terminal to transmit image data, thereby improving the frame selection rate of the terminal in zero-delay photography.

[0204] An embodiment of the present application also provides a chip, which includes a programmable logic circuit and / or program instructions. When the chip is running, it is used to implement the image capture method of the front-end chip or the back-end chip side as described in the above embodiments.

[0205] An embodiment of the present application further provides a computer-readable storage medium, which stores at least one program, and the at least one program is used to be executed by a processor to implement the image capturing method as described in the above embodiment.

[0206] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the image capture method provided in the above embodiment.

[0207] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0208] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image capturing method, characterized in that: The method is used for a front-end chip, and the method comprises: In response to a shooting instruction sent by a back-end chip, a cached image frame is read from a memory, where the cached image frame is output by an image sensor and cached in the memory; Performing scene recognition on the cached image frame to obtain a scene recognition result; Based on the scene recognition result, determining a target frame selection sub-strategy from candidate frame selection sub-strategies, and a strategy execution priority corresponding to the target frame selection sub-strategy; The target frame selection sub-strategy and the strategy execution priority are determined as a target frame selection strategy, wherein the target frame selection strategy is used to indicate the selection of image frames that meet the shooting requirements corresponding to the current shooting scene, and different scenes correspond to different frame selection strategies; Based on the descending order of the policy execution priority, the target frame selection sub-strategy is executed on the cached image frame to obtain a candidate image frame, wherein the number of frames of the candidate image frame is less than the number of frames of the cached image frame; The candidate image frames are transmitted to the back-end chip so that the back-end chip selects a target image frame for generating a photographed image from the candidate image frames.

2. The method according to claim 1, characterized in that In the case where the scene recognition result is a static shooting scene, the target frame selection sub-strategy includes at least a frame selection sub-strategy based on timestamp and a frame selection sub-strategy based on image quality, and the policy execution priority of the frame selection sub-strategy based on image quality is higher than the policy execution priority of the frame selection sub-strategy based on timestamp; When the scene recognition result is a dynamic shooting scene, the target frame selection sub-strategy includes at least a timestamp-based frame selection sub-strategy and an image quality-based frame selection sub-strategy, and the strategy execution priority of the timestamp-based frame selection sub-strategy is higher than the strategy execution priority of the image quality-based frame selection sub-strategy.

3. The method according to claim 1, characterized in that The step of executing the target frame selection sub-strategy on the cached image frame in descending order of the strategy execution priority to obtain a candidate image frame includes: Based on the descending order of the policy execution priority, executing the target frame selection sub-policy of the first part on the cached image frame to obtain the candidate image frame; The method further comprises: The target frame selection strategy is transmitted to the back-end chip so that the back-end chip executes the target frame selection sub-strategy of the second part on the candidate image frame to obtain the target image frame, wherein the target frame selection sub-strategy of the first part is different from the target frame selection sub-strategy of the second part, and the strategy execution priority of the target frame selection sub-strategy of the first part is higher than the strategy execution priority of the target frame selection sub-strategy of the second part.

4. The method according to claim 1, characterized in that: The method further comprises: The scene recognition result and at least one of the target frame selection strategies are transmitted to the back-end chip so that the back-end chip selects the target image frame from the candidate image frames based on the scene recognition result and at least one of the target frame selection strategies.

5. The method according to any one of claims 1 to 4, characterized in that: The buffer of the memory is used to cache n frames of the cached image frames; The step of reading the cached image frame from the memory in response to the shooting instruction sent by the back-end chip includes: In response to the shooting instruction sent by the back-end chip, read n frames of the cached image frames from the memory; The step of executing the target frame selection sub-strategy on the cached image frame in descending order of the strategy execution priority to obtain a candidate image frame includes: Based on the descending order of the policy execution priority, the target frame selection sub-policy is executed on the cached image frame to select the candidate image frame from the n frames of the cached image frames.

6. The method according to any one of claims 1 to 4, characterized in that: In the case where the shooting instruction is not received, the buffer of the memory is used to cache m frames of the cached image frames; The step of reading the cached image frame from the memory in response to the shooting instruction sent by the back-end chip includes: In response to the shooting instruction sent by the back-end chip, read m frames of the cached image frames from the memory; The method further comprises: In response to the shooting instruction sent by the back-end chip, a cache increase instruction is sent to the memory so that the memory increases the capacity of the buffer zone from m frames to n frames; Reading n frames of the cached image frames from the memory; Based on the target frame selection strategy, the candidate image frame is selected from the n cached image frames.

7. The method according to claim 6, characterized in that After reading m frames of the cached image frames from the memory, the method further includes: The target frame selection strategy is determined based on the m frames of cached image frames.

8. An image capturing method, characterized in that: The method is used for a back-end chip, and the method comprises: Sending a shooting instruction to the front-end chip so that the front-end chip reads a cached image frame from the memory, wherein the cached image frame is output by the image sensor and cached in the memory; Receive the candidate image frames transmitted by the front-end chip, the candidate image frames are obtained by the front-end chip executing the target frame selection sub-strategy on the cached image frames based on the descending order of the policy execution priority, wherein the number of frames of the candidate image frames is less than the number of frames of the cached image frames, and the front-end chip is used to perform scene recognition on the cached image frames to obtain scene recognition results; based on the scene recognition results, determine the target frame selection sub-strategy from the candidate frame selection sub-strategies, and the policy execution priority corresponding to the target frame selection sub-strategy; determine the target frame selection sub-strategy and the policy execution priority as the target frame selection strategy, the target frame selection strategy is used to indicate the selection of image frames that meet the shooting requirements corresponding to the current shooting scene, and different scenes correspond to different frame selection strategies; A target image frame for generating a photographic image is selected from the candidate images.

9. The method according to claim 8, characterized in that The method further comprises: receiving at least one of the scene recognition result and the target frame selection strategy transmitted by the front-end chip, wherein the scene recognition result is obtained by the front-end chip performing scene recognition on the cached image frame; The step of selecting a target image frame for generating a photographic image from the candidate images comprises: Based on at least one of the scene recognition result and the target frame selection strategy, the target image frame is selected from the candidate images.

10. The method according to claim 8, characterized in that The target frame selection strategy includes the target frame selection sub-strategy and the strategy execution priority of the target frame selection sub-strategy, and the candidate image frame is obtained by the front-end chip executing the first part of the target frame selection sub-strategy on the cached image frame in descending order of the strategy execution priority; The method further comprises: Receiving the target frame selection strategy transmitted by the front-end chip; The step of selecting a target image frame for generating a photographic image from the candidate images comprises: Based on the descending order of the policy execution priority, the target frame selection sub-strategy of the second part is executed on the candidate image frame to obtain the target image frame, wherein the target frame selection sub-strategy of the first part is different from the target frame selection sub-strategy of the second part, and the policy execution priority of the target frame selection sub-strategy of the first part is higher than the policy execution priority of the target frame selection sub-strategy of the second part.

11. An image capturing device, characterized in that: The device is used for a front-end chip, and the device comprises: A reading module, configured to read a cached image frame from a memory in response to a shooting instruction sent by a back-end chip, wherein the cached image frame is output by an image sensor and cached in the memory; A first frame selection module, used to perform scene recognition on the cached image frame to obtain a scene recognition result; Based on the scene recognition result, determining a target frame selection sub-strategy from candidate frame selection sub-strategies, and a strategy execution priority corresponding to the target frame selection sub-strategy; The target frame selection sub-strategy and the strategy execution priority are determined as a target frame selection strategy, wherein the target frame selection strategy is used to indicate the selection of image frames that meet the shooting requirements corresponding to the current shooting scene, and different scenes correspond to different frame selection strategies; Based on the descending order of the policy execution priority, the target frame selection sub-strategy is executed on the cached image frame to obtain a candidate image frame, wherein the number of frames of the candidate image frame is less than the number of frames of the cached image frame; The transmission module is used to transmit the candidate image frames to the back-end chip so that the back-end chip selects a target image frame for generating a photographed image from the candidate image frames.

12. An image capturing device, characterized in that: The device is used for a back-end chip, and the device comprises: A sending module, used for sending a shooting instruction to the front-end chip, so that the front-end chip reads a cached image frame from the memory, wherein the cached image frame is output by the image sensor and cached in the memory; A receiving module, used for receiving the candidate image frames transmitted by the front-end chip, wherein the candidate image frames are obtained by the front-end chip executing a target frame selection sub-strategy on the cached image frames based on the descending order of the policy execution priority, wherein the number of frames of the candidate image frames is less than the number of frames of the cached image frames, and the front-end chip is used for performing scene recognition on the cached image frames to obtain a scene recognition result; based on the scene recognition result, determining the target frame selection sub-strategy from the candidate frame selection sub-strategies, and the policy execution priority corresponding to the target frame selection sub-strategy; determining the target frame selection sub-strategy and the policy execution priority as the target frame selection strategy, wherein the target frame selection strategy is used to indicate the selection of image frames that meet the shooting requirements corresponding to the current shooting scene, and different scenes correspond to different frame selection strategies; The second frame selection module is used to select a target image frame for generating a photographed image from the candidate images.

13. A chip, characterized in that: The chip includes a programmable logic circuit and / or program instructions, and when the chip is running, it is used to implement the image capturing method as described in any one of claims 1 to 7, or to implement the image capturing method as described in any one of claims 8 to 10.

14. A terminal, characterized in that: The terminal comprises a front-end chip, a back-end chip and a memory, wherein the front-end chip is used to load and execute a program stored in the memory to implement the image capturing method according to any one of claims 1 to 7; The back-end chip is used to load and execute the program stored in the memory to implement the image capturing method as described in any one of claims 8 to 10.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the image capturing method according to any one of claims 1 to 10.

16. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the image capture method as described in any one of claims 1 to 10.