Shooting method and electronic equipment
By identifying wonderful frames in the preview stream in the electronic device and generating high-resolution photo frames, the problem of low quality of camera preview stream in the prior art is solved, high-quality video recording is achieved, and user operations is simplified, and shooting experience is enhanced. .
Patent Information
- Application Number
- CN202310748245.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-21
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-06-21
AI Technical Summary
The prior art is difficult to effectively capture and record exciting moments in the camera preview stream, resulting in low quality of photos and cumbersome user operations, affecting the shooting experience.
By identifying wonderful frames in the preview stream in the electronic device, generating high-resolution photo frames, and inserting image frames into the video, forming a clear and vivid video stream, reducing user manual operations.
Improves the image quality of photos and videos, simplifies user operations, enhances the shooting experience, can automatically capture and record exciting moments, and generates smoother and more coherent videos.
Smart Images

Figure CN117692762B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of terminal technology, and in particular to a photographing method and an electronic device. Background Art
[0002] There are often some beautiful and wonderful moments in daily life. People will use electronic devices with cameras to take photos or videos to record beautiful and wonderful moments. After the user points the camera at the target scene, the electronic device will display the preview screen of the target scene on the display screen in real time. When the user presses the shutter button, the electronic device can generate a photo corresponding to the preview screen and store the photo. Summary of the invention
[0003] The present application provides a shooting method and an electronic device. When the camera is aimed at a target scene, the electronic device can analyze the preview stream, and then the electronic device can use a wonderful frame recognition method to identify multiple wonderful frames from the preview stream. Then the electronic device can generate a photo frame corresponding to the wonderful frame, and perform frame insertion processing on the photo frame. Finally, the electronic device can use the photo frame corresponding to the wonderful frame and the image frame inserted in the above-mentioned photo frame to generate a clear and vivid video.
[0004] In a first aspect, the present application provides a shooting method, which is applied to an electronic device, wherein the electronic device includes a camera, and the method includes: the electronic device uses a preview stream to display a preview screen on a display screen, and the preview stream includes multiple preview frames; the electronic device identifies multiple wonderful frames from the multiple preview frames, and the moving objects in the wonderful frames are in a preset motion; the electronic device obtains the original images corresponding to each of the multiple wonderful frames; the electronic device processes the original images corresponding to each of the multiple wonderful frames to generate multiple photo frames, and the resolution of the photo frames is greater than the resolution of the preview frames; the electronic device saves the video to a gallery, and the video includes the multiple photo frames.
[0005] After the camera in the electronic device is aimed at the target scene, the electronic device will display a preview screen of the target scene on the display screen through the preview stream. The preview screen is displayed after the electronic device sends the preview frame in the preview stream. The preview stream may include multiple preview frames, and the electronic device can identify the wonderful frame from the preview frames in the preview stream. The moving object in the image content of the wonderful frame is in a preset motion. The moving object can be a person, an animal, a plant or other inanimate objects, etc. The electronic device can obtain the original image corresponding to the wonderful frame according to the index of the wonderful frame. The original image is an image captured by the camera and not processed by the processor. The original image corresponding to the wonderful frame means that the original image can be processed by the processor (such as an image signal processor) to obtain the wonderful frame. After the electronic device obtains the original image corresponding to the wonderful frame, it can process the original image corresponding to the wonderful frame to generate a photo frame corresponding to the wonderful frame. It should be noted that the electronic device can obtain multiple wonderful frames through the preview stream, and the electronic device can obtain multiple photo frames after processing the multiple wonderful frames. Since the electronic device uses a downsampling algorithm when processing the original image to generate the preview frame, the resolution of the preview frame is low. When generating a photo frame, the electronic device uses more image enhancement algorithms than when generating a preview frame, which makes the photo frame have a higher resolution and higher image quality than the preview frame. The electronic device can generate a video based on the photo frames corresponding to multiple wonderful frames and save the video to the gallery.
[0006] The electronic device can extract multiple wonderful frames after the user points the camera at the target scene. Then the electronic device can generate photo frames corresponding to the wonderful frames, and generate videos based on the photo frames. The user can obtain vivid and clear videos without manually clicking the shutter button, which can reduce the tediousness of user operations and improve the user's shooting experience.
[0007] In combination with the first aspect, in some embodiments, the video also includes one or more inserted frames between multiple photographic frames. In the inserted frames between two consecutive photographic frames, the position of the moving object falls on a first motion trajectory, and the first motion trajectory is a motion trajectory formed by the pixel movement of the moving object in two consecutive photographic frames.
[0008] Among them, in the video generated by the electronic device according to the photo frame corresponding to the wonderful frame, in addition to multiple photo frames, it also includes one or more image frames inserted by the electronic device between the photo frames. The image frame inserted by the electronic device between the photo frames can also be called an insertion frame. When generating the insertion frame, the electronic device can also use the image enhancement algorithm to process the insertion frame. Since the camera in the electronic device also saves the timestamp of each original image when collecting the original image, the electronic device can arrange the photo frames in chronological order according to the timestamp of the original image corresponding to the photo frame. Then the electronic device can insert one or more insertion frames between adjacent photo frames. Exemplarily, the electronic device can determine the motion trajectory formed by the pixel movement of the moving object in two consecutive photo frames, and the above motion trajectory can be called the first motion trajectory. Then the electronic device can generate multiple insertion frames, and the moving object is on the first motion trajectory in the image of the insertion frame. And the insertion frames can be arranged from near to far from the starting point of the first motion trajectory according to the position of the moving object in the content of the insertion frame image.
[0009] That is, the electronic device can generate multiple interpolated frames according to the motion trajectory of the moving object in the image of the photographed frame. Furthermore, the electronic device can generate a video according to the photographed frame and the multiple interpolated frames. Since the position of the moving object in the interpolated frame can reflect the state of the moving object between adjacent photographed frames, the video generated by the electronic device according to the photographed frame and the interpolated frame will be more coherent and smooth.
[0010] In combination with the first aspect, in some embodiments, the number of inserted frames between every two adjacent photographing frames is the same.
[0011] That is to say, the electronic device can insert the same number of insertion frames between every two adjacent photographic frames. In this way, in the video generated by the electronic device, the time length of the insertion frames transitioning between every two photographic frames corresponding to the wonderful frames is equal.
[0012] In combination with the first aspect, in some embodiments, the multiple photo frames include an adjacent first photo frame and a second photo frame, an adjacent third photo frame and a fourth photo frame, K1 insertion frames are inserted between the first photo frame and the second photo frame, and K2 insertion frames are inserted between the third photo frame and the fourth photo frame, where K1 and K2 are positive integers, and K1 is less than K2.
[0013] That is to say, the number of inserted frames inserted by the electronic device between every two adjacent photographic frames may be different. Thus, in the video generated by the electronic device, the time length of the inserted frames transitioning between every two photographic frames corresponding to the wonderful frames is not equal.
[0014] In combination with the first aspect, in some embodiments, the third photo frame and the fourth photo frame are more exciting than the first photo frame and the second photo frame.
[0015] The electronic device can determine the wonderfulness of each wonderful frame according to the image content of each wonderful frame. The electronic device can insert more insertion frames between the photo frames corresponding to the wonderful frames with higher wonderfulness, so that the transition time of the insertion frames between the above two photo frames is longer, and the transition between the two photo frames in the video will be smoother. The user can watch a video with more focus.
[0016] In combination with the first aspect, in some embodiments, the electronic device identifies multiple wonderful frames from multiple preview frames, including: the electronic device identifies key points of the moving object in the multiple preview frames, the key points of the moving object are used to represent the posture of the moving object; the electronic device can identify multiple wonderful frames based on the key points of the moving object, and the posture of the moving object in the wonderful frames matches the first template.
[0017] The electronic device can identify the key points of the moving object in the preview frame. The key points of different types of moving objects may be different. For example, for humans and animals, the key points may be the skeleton points of humans and animals, and the key points of inanimate objects may be one or more points on the outline of the object. The electronic device may store a first template, which includes the key points of the moving object under a specific motion. The electronic device may match the key points of the moving object in the preview frame with the key points of the moving object in the first template under the specific motion. For example, the electronic device may calculate the similarity between the key points of the moving object in the preview frame and the key points of the moving object in the first template under the specific motion. When the similarity is higher than a first threshold, the electronic device may identify the preview frame as a wonderful frame. Optionally, the electronic device may also store a second threshold, and when the similarity is higher than the second threshold, the electronic device may confirm that the wonderful degree of the wonderful frame is high. The electronic device may insert a larger number of image frames between the photo frames corresponding to the wonderful frames with a higher degree of wonderfulness.
[0018] The first template may be selected by the user. Figure 5C In the illustrated embodiment, the electronic device may receive a user's selection operation on a shooting scene option, and in response to the operation, the electronic device may select a first template corresponding to the shooting scene option. Alternatively, the first template may be selected by the electronic device from a plurality of templates based on the first shooting scene, and each of the plurality of templates corresponds to a different shooting scene, and the first shooting scene may be a shooting scene presented in the preview screen. The electronic device may identify the shooting scene after the camera is aimed at the target shooting scene. Then, the electronic device may select the first template corresponding to the shooting scene to identify the wonderful frames in the preview frame.
[0019] In combination with the first aspect, in some embodiments, the electronic device identifies multiple wonderful frames from multiple preview frames, including: the electronic device determines a text description of each of the multiple preview frames, the text description is used to describe the content expressed by the preview frames; the electronic device selects multiple wonderful frames from the multiple preview frames according to the text description, there is a first keyword in the text description of the multiple wonderful frames, and the first keyword is used to describe a preset motion.
[0020] The electronic device can recognize the text description of the preview frame, and the above text description is used to describe the content expressed by the image of the preview frame. Among them, the electronic device can recognize the text description of the preview frame by a machine learning method. One or more keywords corresponding to the wonderful frame can be obtained in the electronic device, including a first keyword. When the text description of the preview frame contains the first keyword, the electronic device can identify the preview frame as a wonderful frame. Among them, the first keyword can describe a preset movement of a moving object, such as "jumping", "shooting", "blowing candles", etc.
[0021] In conjunction with the first aspect, in some embodiments, the electronic device can also use the preview frames between two wonderful frames as wonderful frames. That is, after the electronic device identifies the wonderful frames by using key points of the moving objects in the preview frames or text descriptions of the preview frames, it can also use the preview frames between the two wonderful frames as wonderful frames.
[0022] In combination with the first aspect, in some embodiments, the electronic device can also use the first Y1 preview frames and / or the last Y2 preview frames adjacent to the wonderful frame as wonderful frames, where Y1 and Y2 are positive integers. That is, after the electronic device identifies the wonderful frame by using the key points of the moving object in the preview frame or the text description of the preview frame, it can also use the preview frames adjacent to the wonderful frame before and after as wonderful frames.
[0023] In this way, it can be avoided that when the electronic device identifies the wonderful frames in the preview frames, the number of wonderful frames recognized by the electronic device is too small, resulting in poor effect of inserting the photo frames corresponding to the wonderful frames, due to the fact that the templates corresponding to the wonderful frames are not comprehensive enough, or the keywords of the text descriptions corresponding to the wonderful frames are not comprehensive enough, thereby affecting the smoothness of the video generated by the electronic device.
[0024] In combination with the first aspect, in some embodiments, the electronic device displays a preview image on a display screen using a preview stream and also displays a highlight capture button, and before the electronic device identifies multiple highlight frames from multiple preview frames, the electronic device further includes: the electronic device detects an operation on the highlight capture button. That is, the electronic device can receive and respond to the user's operation on the highlight capture button, and then identify the highlight frames in the preview stream.
[0025] In a second aspect, the present application provides an electronic device, comprising a display screen, a memory, and a processor coupled to the memory; the display screen is used to display an interface, the memory stores a computer program, and when the processor executes the above computer program, the electronic device implements any one of the methods described in the above first aspect.
[0026] In a third aspect, the present application provides a computer-readable storage medium, which stores a computer program or computer instructions, and the aforementioned computer program or computer instructions are executed by a processor to implement any method described in the first aspect above.
[0027] In a fourth aspect, an embodiment of the present application provides a computer program product. When the computer program product is executed by a processor, the method described in any one of the above-mentioned first aspects will be implemented.
[0028] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a memory, wherein the memory is used to store computer programs or computer instructions, and the processor is used to execute the computer programs or computer instructions stored in the memory, so that the chip executes any method described in the first aspect above.
[0029] The solutions provided in the second to fifth aspects are used to implement or cooperate with the corresponding methods provided in the first aspect, and therefore can achieve the same or corresponding beneficial effects as the corresponding methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 is a schematic diagram of the architecture of an electronic device 100 provided in an embodiment of the present application;
[0031] Figure 2 is a software structure block diagram of the electronic device 100 provided in an embodiment of the present application;
[0032] Figure 3A is a flow chart of a shooting method provided by an embodiment of the present application;
[0033] Figure 3B-Figure 3C It is a schematic diagram of a process in which an electronic device according to an embodiment of the present application identifies a wonderful frame from a preview stream and generates a photographing frame corresponding to the wonderful frame;
[0034] Figure 3D-Figure 3F This is a schematic diagram of a process of inserting a photographed frame by an electronic device provided in an embodiment of the present application;
[0035] Figure 4 is a flow chart of a method for some electronic devices to identify wonderful frames provided by an embodiment of the present application;
[0036] Figure 5A-5JIt is a series of user interaction schematic diagrams involved in the shooting method provided in the embodiment of the present application. DETAILED DESCRIPTION
[0037] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.
[0038] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.
[0039] In daily life, people often use electronic devices with cameras to take photos to record a moment in a wonderful clip. Sometimes users may feel that the photos are static and not vivid enough, and hope that the electronic device can take high-quality photos and reproduce the above wonderful clip more vividly.
[0040] The embodiments of the present application provide a shooting method and an electronic device. The electronic device can identify one or more wonderful frames through a preview stream, and then determine the original image corresponding to the wonderful frame. The electronic device can process the original image to generate a high-definition image with higher image quality. Then, the electronic device can insert a new image frame into the high-definition image and generate a video. Compared with photos, the video can more vividly present a series of wonderful moments of the subject being photographed, thereby improving the user's shooting experience.
[0041] The following first introduces the electronic device 100 provided in an embodiment of the present application.
[0042] Figure 1 It is a schematic diagram of the architecture of the electronic device 100 provided in an embodiment of the present application.
[0043] The electronic device 100 may be equipped with Or portable terminal devices with other operating systems, such as mobile phones, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPC), netbooks, as well as cellular phones, personal digital assistants (PDA), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, vehicle-mounted devices, smart home devices and / or smart city devices, etc.
[0044] like Figure 1 As shown, the electronic device 100 may include a camera 110, an image signal processor (ISP) 111, an application processor (AP) 112, a buffer memory 113, a display screen 114, an encoder 115, an encoder 116, and an external memory 117, and these components are coupled through one or more groups of buses. The above-mentioned bus may be an inter-integrated circuit (I2C) bus, an inter-integrated circuit sound (I2S) bus, a pulse code modulation (PCM) bus, a mobile industry processor interface (MIPI, etc.).
[0045] The camera 110 may include: a lens (not shown in the figure), a photosensitive sensor and a flexible printed circuit board (FPCB) part (not shown in the figure). The FPCB is responsible for connecting other components of the camera 110, such as the photosensitive sensor, to the image signal processor (ISP) 111, for example, transmitting the raw data output by the photosensitive sensor to the ISP. When taking a picture, the shutter of the camera 110 is opened, and light enters and shines on the photosensitive sensor. The photosensitive sensor converts the light signal into an electrical signal, and then further converts the electrical signal into a digital signal through analog digital convert (ADC) to pass it to the ISP for processing. The data of the digital signal is the original image data collected by the camera, and its data format may be, for example, a Bayer arrangement format. The original image data is also called a RAW image.
[0046] The ISP may be used to perform a series of image processing on the RAW image to obtain a YUV frame or an RGB frame. The series of image processing may include: automatic exposure control (AEC), automatic gain control (AGC), automatic white balance (AWB), color correction, bad pixel removal, etc. The ISP may also be integrated into the camera 110.
[0047] The display screen can be used to display images captured by the camera, such as preview images (preview frames). The preview image can be a YUV frame or RGB frame output by the ISP, which is further obtained through algorithms such as downsampling. Its clarity is often lower than that of a photo, so as to avoid a long delay in displaying the preview frame on the display screen due to the high clarity of the preview frame. A series of preview images (preview frames) are arranged in chronological order to form a preview stream. Based on the preview stream, the display screen can present the images captured by the camera in real time. The preview stream needs to be sent to the display screen before it can be displayed on the display screen. Sending to the display screen means pushing the preview image captured by the camera to the frame buffer (FB) for storage. The frame buffer is a storage space that can be located in the video memory or in the internal memory, and is used to store the rendering data processed by the graphics chip or to be extracted. The content of the frame buffer corresponds to the interface display on the display screen, which can be simply understood as the cache corresponding to the content displayed on the display screen. In other words, modifying the content in the frame buffer is to modify the picture displayed on the display screen.
[0048] The buffer memory 113 may be used to cache the RAW image output by the photosensor.
[0049] The encoder 115 may be used to encode the YUV frame or RGB frame output by the ISP to obtain a photo. The format of the photo output by the encoder 115 may include but is not limited to: any format such as Joint Photographic Experts Group (JPEG), Tag Image File Format (TIFF), etc.
[0050] The encoder 116 may be used to encode the YUV frame or RGB frame output by the ISP to obtain a video. The format of the video output by the encoder 116 may include, but is not limited to, moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc. In some embodiments, the encoder 115 and the encoder 116 may be different encoding units of the same encoder. Alternatively, the encoder 115 and / or the encoder 116 may be encoding units integrated in an application processor.
[0051] The application processor (AP) 112 can be coupled to one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the application processor, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data, etc. The non-volatile memory can also store executable programs and store user and application data, etc., and can be loaded into the random access memory in advance for direct reading and writing by the application processor. A storage unit can also be set in the application processor, and the storage unit can be a high-speed cache storage unit, which can be used to save instructions or data that the application processor has just used or circulated. The implementation code of the shooting method provided in the embodiment of the present application can be stored in the NVM. After starting the camera application, the code can be loaded into the RAM. In this way, the application processor can read the program code directly from the RAM to implement the shooting method provided in the embodiment of the present application.
[0052] The application processor 112 can be used to identify multiple wonderful frames from the preview stream, and send the index of the wonderful frame to the ISP, triggering the ISP to take out the RAW image corresponding to the wonderful frame from the buffer memory. The application processor can process the above RAW image to generate a photo frame corresponding to the wonderful frame. The photo frame can be obtained by processing the YUV frame or RGB frame output by the ISP through the image enhancement algorithm. The resolution of the photo frame is higher than the resolution of the wonderful frame, which can better present the details of the wonderful moment. The application processor can also be used to perform interpolation processing on the photo frame corresponding to the wonderful frame output by the ISP, and further output it to the encoder 116 to trigger the encoder 116 to perform video encoding on the multiple frames of images after interpolation to obtain a video. It should be noted that the video is composed of multiple image frames. A frame (Frame) image in the video presents a still picture. A frame image can be composed of multiple pixels. The more pixels a frame image has, the higher the resolution, the clearer the image, and the larger the storage space occupied by the image. Since the application processor performs interpolation processing on the photographed frames when generating the video, the above video can present a more continuous and wonderful picture; and since the application processor uses photographed frames with higher definition to generate the video, the definition of the above video is very high.
[0053] In addition, the external memory 117 can be a type of NVM, which can be used to store image files such as photos and videos. Photos and videos can be specifically saved in a path accessible to the gallery application, so that the user can view the photos and videos in the path when opening the gallery. The gallery is an application that manages image files such as photos and videos, and can also be named as an album.
[0054] Figure 1 The schematic structure does not constitute a specific limitation on the electronic device 100, and the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. For example, the electronic device may also include a graphics processing unit (GPU) for rendering. For another example, the electronic device 100 may also include a variety of sensors: pressure sensors, distance sensors, proximity light sensors, touch sensors, ambient light sensors, etc. Among them, the touch sensor may also be referred to as a "touch panel" and may be provided on a display screen. The touch sensor and the display screen may form a touch screen, also known as a "touch screen". The touch sensor is used to detect a touch operation acting thereon, which may be an operation of a user pressing a shutter key. The touch operation detected by the touch sensor may be passed to an application processor (AP) to determine the type of touch event.
[0055] Figure 2 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.
[0056] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the system is divided into four layers, from top to bottom: application layer, application framework layer, runtime and system library, and kernel layer.
[0057] The application layer can include a series of application packages.
[0058] like Figure 2 As shown, the application package may include camera, gallery, calendar, call, map, navigation, music, video and other applications (also referred to as apps).
[0059] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.
[0060] like Figure 2 As shown, the application framework layer may include a camera access interface, a window manager, a content provider, a view system, a phone manager, a resource manager, and the like.
[0061] The camera access interface can be used to provide an application programming interface and a programming framework for the camera application. The camera access interface can include a camera manager, and the camera manager can be used to implement one or more methods for the electronic device to generate a video in the above embodiments.
[0062] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.
[0063] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.
[0064] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.
[0065] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).
[0066] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.
[0067] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications from applications running in the background, or a notification that appears on the screen in the form of a dialogue interface. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.
[0068] The runtime includes the core library and the virtual machine. The runtime is responsible for the scheduling and management of the system.
[0069] The system library consists of two parts: one part is the function that the programming language (for example, Java language) needs to call, and the other part is the core library of the system.
[0070] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the programming files (e.g., java files) of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
[0071] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), camera algorithm library, etc.
[0072] The surface manager is used to manage the display subsystem and provides the fusion of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications.
[0073] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
[0074] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.
[0075] A 2D graphics engine is a drawing engine for 2D drawings.
[0076] The camera algorithm library may include one or more image or video processing algorithms. For example, the camera algorithm library may include one or more image processing algorithms when the ISP processes the RAW image to generate a preview stream and a photo frame. In addition, the camera algorithm library may also include an algorithm for the application processor to identify outstanding frames from the preview stream, an algorithm for the application processor to interpolate the photo frame corresponding to the outstanding frame, etc.
[0077] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver. The camera driver can be used to drive the camera sensor to collect images and drive the ISP and application processor to process images.
[0078] In some embodiments, when the touch sensor receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is a control of the camera application icon, the camera application calls the camera access interface of the application framework layer, starts the camera application, and then starts the camera driver by calling the kernel layer to capture the RAW image through the camera. In response to the user operation, the camera application can call the camera driver through the camera access interface, and then the camera driver can drive the ISP or application processor to implement one or more algorithms in the above-mentioned camera algorithm library.
[0079] Based on the software and hardware architecture of the electronic device 100 introduced above, the shooting method provided in the embodiment of the present application is introduced below.
[0080] Figure 3A The process of the shooting method provided by the embodiment of the present application is shown. The method may include:
[0081] S301: The electronic device displays a preview picture on a display screen using a preview stream, where the preview stream includes multiple preview frames.
[0082] The electronic device 100 can send the preview frame to display, so that the preview screen corresponding to the preview stream will be displayed on the display screen. The method for displaying the preview screen by the electronic device 100 can refer to Figure 1 The description of the illustrated embodiment will not be repeated here. Since the preview image needs to be displayed on the display screen in real time, the resolution of the preview stream is relatively low.
[0083] S302: The electronic device identifies a plurality of wonderful frames from a plurality of preview frames, wherein the moving objects in the wonderful frames are in a preset motion.
[0084] The electronic device 100 can read the preview frames in the preview stream in sequence, and then identify the wonderful frames in the preview stream. Wonderful frames refer to preview frames with more wonderful image content, for example, the subject in the image content is in a preset motion. The above-mentioned subject can be an object in motion. The above-mentioned moving object can be a person, and the preset motion can be, for example, the motion of an athlete jumping up to shoot a basketball, or the motion of a person blowing out birthday candles. Not limited to people, the moving object can also be an animal, a plant, or even an inanimate object (such as a curtain, a firework), etc. The preset motion can be, for example, the motion of a cat jumping up onto a table, the motion of reeds being blown by a strong wind, the motion of fireworks blooming in the air, and so on.
[0085] The electronic device 100 can determine that the moving object is in a preset motion by identifying key points of the moving object in the preview frame or identifying text descriptions of the preview frame image content. The specific method for the electronic device 100 to determine that the moving object is in a preset motion can be referred to the description of the subsequent embodiments, which will not be expanded here.
[0086] S303: The electronic device obtains original images corresponding to the multiple wonderful frames.
[0087] The original image is the RAW image output by the photosensitive sensor. The original image corresponding to the wonderful frame refers to the original image used to generate the wonderful frame. The electronic device 100 will mark the index of each RAW image, and the above index can reflect the time sequence of the RAW image output by the photosensitive sensor. The index of the RAW image stored in the electronic device 100 can be the timestamp of the image sensor outputting the RAW image.
[0088] The electronic device 100 also marks the index of each preview frame, wherein the index of the preview frame corresponds to the index of the RAW image that generates the preview frame. In this way, the ISP 111 can fetch the RAW image corresponding to the wonderful frame from the buffer memory 113 according to the index of the wonderful frame.
[0089] S304: The electronic device processes the original images corresponding to the multiple wonderful frames to generate multiple photo frames.
[0090] After the electronic device 100 obtains the RAW image corresponding to the wonderful frame, it can process the RAW image to generate multiple photo frames. The method for the electronic device 100 to generate multiple photo frames can be referred to Figure 1 The description of the illustrated embodiment will not be repeated here.
[0091] S305: The electronic device saves the video to a gallery, where the video includes multiple photo frames.
[0092] After the electronic device 100 generates multiple photo frames, one or more image frames may be inserted between the multiple photo frames. The image frames inserted by the electronic device 100 may be generated based on the image content of two or more adjacent frames. The method by which the electronic device generates the above-mentioned image frames may include but is not limited to: copying adjacent frames, mixing adjacent frames, or optical flow interpolation method, etc. In this application document, the image frames inserted by the electronic device between the photo frames may also be referred to as inserted frames.
[0093] Taking two adjacent frames, A frame and B frame, as an example, the interpolation method of copying adjacent frames may refer to copying one or more A frames and / or one or more B frames, and then inserting the copied image frames between the A frame and the B frame. The interpolation method of mixing adjacent frames may be to assign an A weight to the A frame, assign a B weight to the B frame, and then multiply each pixel in the A frame by the A weight, and multiply each pixel in the B frame by the B weight. Then, the result of multiplying the pixel at each position in the A frame by the weight is added to the result of multiplying the pixel at the corresponding position in the B frame by the weight, to obtain the pixel value at each position of the image frame to be inserted between the A frame and the B frame. For the introduction of the optical flow interpolation method, please refer to the introduction of the subsequent embodiments, which will not be expanded here.
[0094] In some embodiments, the electronic device 100 may determine the degree of excellence of the wonderful frame. When the degree of excellence of the wonderful frame is high, the electronic device 100 may insert a larger number of image frames between the photo frame corresponding to the wonderful frame and its adjacent photo frame. The method for the electronic device 100 to determine the degree of excellence of the wonderful frame can be referred to the introduction of the subsequent embodiments, which will not be expanded here.
[0095] After the electronic device inserts one or more image frames between the multiple camera frames, the encoder 116 can process the multiple camera frames and the one or more image frames to generate a video. The electronic device 100 can save the video in a path accessible to the gallery application so that the user can view the video when opening the gallery.
[0096] By implementing Figure 3A With the shooting method shown, the electronic device 100 can shoot a wonderful video for the user. Compared with static photos, videos can reproduce a series of wonderful moments of the subject more vividly and vividly. In addition, the electronic device 100 can identify wonderful moments when the user previews, without the user manually triggering the shooting, which can avoid missing these wonderful moments due to delayed user reaction. Finally, since the above-mentioned wonderful video is generated by the photo frames, and the electronic device also inserts new image frames between adjacent photo frames, the finally generated wonderful video is clearer and smoother.
[0097] Figure 3B The example shows a process in which an electronic device identifies a wonderful frame from a preview stream and generates a photographing frame corresponding to the wonderful frame.
[0098] Figure 3BThere are three types of labels in total: "Photo Stream", "RAW Image" and "Preview Stream". For ease of understanding, one or more blocks under the "RAW Image" label represent one or more RAW images output by the photosensitive sensor, and the number in the block below the "RAW Image" label represents the index of the RAW image. Similarly, one or more blocks under the "Photo Stream" label represent one or more photo frames. The number in the block below the "Photo Stream" label represents the index of the photo frame. One or more blocks under the "Preview Stream" label represent one or more preview frames. The number in the block below the "Preview Stream" label represents the index of the preview frame. Among them, the index of the RAW image, the index of the preview frame and the index of the photo frame correspond to each other. That is to say, the preview frame with an index of 2 and the photo frame with an index of 2 are obtained after the electronic device processes the RAW image with an index of 2. It should be noted that the indexes of the photo frames, RAW images and preview frames in the electronic device can be in other forms (such as timestamps), and the embodiments of the present application are not limited to this.
[0099] The electronic device 100 may identify N wonderful frames through the preview stream, including but not limited to:
[0100] 1. The photosensitive sensor in the electronic device 100 outputs P RAW images, where P is a positive integer.
[0101] 2. The electronic device 100 processes the P RAW images to generate P preview frames.
[0102] The photosensitive sensor in the electronic device 100 can continuously output RAW images after exposure. The photosensitive sensor outputs a total of P RAW images, where P is a positive integer. Then the electronic device can process the P RAW images in sequence to generate P preview frames. The method for the electronic device to process the RAW images to generate preview frames can be referred to Figure 1 The description of the illustrated embodiment will not be repeated here.
[0103] 3. The electronic device 100 determines N highlight frames among the P preview frames.
[0104] 4. The electronic device processes the RAW images corresponding to the N wonderful frames to generate photographic frames corresponding to the wonderful frames.
[0105] The electronic device 100 can determine one or more wonderful frames through the preview stream. Assuming that N preview frames with indexes 2 to P are determined as wonderful frames, the electronic device can take out N RAW images with indexes 2 to P corresponding to the wonderful frames, and then process the N RAW images to generate photo frames corresponding to the wonderful frames.
[0106] like Figure 3CAs shown, assuming that the preview frame with index 2 in the preview stream is identified as a wonderful frame by the wonderful frame recognition algorithm, the RAW image corresponding to the preview frame with index 2 will be taken out. The electronic device can process the RAW image to generate a photo frame. After the electronic device 100 processes the preview frame with index 2, it can continue to read the preview frame with index 3. Then the electronic device 100 can use the wonderful frame recognition algorithm to identify the preview frame with index 3. The electronic device 100 can use the wonderful frame recognition algorithm to identify the preview frames in the preview stream in sequence until there are no preview frames in the preview stream that have not been identified by the wonderful frame recognition algorithm. It should be noted that the electronic device identifies the wonderful frames and converts the RAW images into preview frames synchronously. That is to say, when the electronic device uses the wonderful frame recognition algorithm to identify the preview frames in the preview stream, new preview frames will be added to the preview stream. The method for the electronic device to generate a photo frame based on the RAW image can refer to the introduction of the aforementioned embodiment, which will not be repeated here.
[0107] Figure 3D The process of inserting frames of the electronic device into corresponding photographing frames of the wonderful frames is exemplified. Figure 3D The embodiment shown can be regarded as Figure 3B In the illustrated embodiment, after the electronic device generates a wonderful frame corresponding to a photographing frame, the electronic device further processes the photographing frame.
[0108] like Figure 3D As shown, the electronic device 100 can insert one or more image frames between two adjacent photo frames. The image frames inserted by the electronic device 100 can be generated according to the image contents of the adjacent photo frames. For example, the image frames inserted between the photo frame with index 2 and the photo frame with index 3 can be generated by the image contents of the photo frame with index 2 and the image contents of the photo frame with index 3.
[0109] Figure 3E The process of using the optical flow interpolation method to interpolate frames by an electronic device is exemplified. Figure 3D Take the photo frame with index 2 and the photo frame with index 3 as an example. Figure 3E As shown in step 1 in, the electronic device 100 can input the photo frame with index 2 and the photo frame with index 3 into the neural network model A, and the neural network model A can output a mapping table from the photo frame with index 2 to the photo frame with index 3. The mapping table can determine the position of the photographed object in the photo frame with index 2 in the photo frame with index 3. In this way, the electronic device 100 can determine the motion trajectory of the photographed object between the two photo frames according to the mapping table (for example, the electronic device 100 can determine that the photographed object has moved from position A to position B). Among them, the electronic device 100 determines the motion trajectory of the photographed object between two adjacent photo frames by the pixel movement of the photographed object between the two photo frames.
[0110] like Figure 3E As shown in step 2 in, the electronic device 100 can input the photographic frame with index 2, the photographic frame with index 3 and the above-mentioned mapping table into the neural network model B. The neural network model B can generate K1 new image frames according to the adjacent photographic frames and the mapping tables corresponding to the adjacent photographic frames, and insert the K1 new image frames between the photographic frame with index 2 and the photographic frame with index 3. Among them, the position of the photographed object in the newly generated image frame is on the motion trajectory of the photographed object between the photographic frame with index 2 and the photographic frame with index 3. According to the order in which the electronic device inserts the image frames, the object in the image frame gradually moves away from position A and approaches position B.
[0111] The neural network model A and the neural network model B may be different neural network models. The types of the neural network models may include but are not limited to convolutional neural network (CNN) models, recurrent neural network (RNN) models, long short-term memory (LSTM) models, and deep neural network (DNN) models.
[0112] The method for the electronic device 100 to insert multiple image frames into other adjacent frames may refer to the method for the electronic device 100 to insert K1 image frames into the photo frame with index 2 and the photo frame with index 3, which will not be described in detail here.
[0113] The number of image frames inserted by the electronic device between every two adjacent photo frames may be the same. Figure 3F As shown, the photographic frame with index 2 and the photographic frame with index 3 are two adjacent photographic frames, the photographic frame with index 3 and the photographic frame with index 4 are two adjacent photographic frames, ..., the photographic frame with index P-1 and the photographic frame with index P are two adjacent photographic frames. The number of image frames inserted between the photographic frame with index 2 and the photographic frame with index 3, between the photographic frame with index 3 and the photographic frame with index 4, ..., between the photographic frame with index P-1 and the photographic frame with index P can all be Q1, where Q1 is a positive integer. In this way, when the subsequent electronic device generates a video from the photographic frames and the image frames, the time length between each two adjacent photographic frames in the video is equal. For example, when generating a video, the electronic device 100 can set the duration of each frame displayed in the video to 3 milliseconds. If the electronic device 100 inserts 30 image frames between each two adjacent photographic frames, the time between each two photographic frames in the video is 90 milliseconds.
[0114] Without limitation, the number of image frames inserted by the electronic device between each two adjacent photo frames may be different. For example, the electronic device may insert Q1 image frames between the photo frame with index 2 and the photo frame with index 3, and insert Q2 image frames between the photo frame with index 3 and the photo frame with index 4. Wherein, Q1 and Q2 are positive integers, and Q2 is greater than Q1. In this way, in the video generated by the electronic device according to the photo frame and the image frame inserted in the photo frame, the time length of different adjacent photo frames played may not be equal. For example, when generating a video, the electronic device 100 may set the time length of each frame displayed in the video to 3 milliseconds. The electronic device may insert 30 image frames between the photo frame with index 2 and the photo frame with index 3, and insert 50 image frames between the photo frame with index 3 and the photo frame with index 4. In this way, in the video generated by the electronic device 100, a total of 90 milliseconds pass from the photo frame with index 2 to the photo frame with index 3, and a total of 150 milliseconds pass from the photo frame with index 3 to the photo frame with index 4. The number of image frames inserted by the electronic device 100 in the photo frame and the playback time of each frame in the video generated by the electronic device 100 are only examples and are not limitations of the embodiments of the present application. It should be noted that when the electronic device uses the optical flow interpolation method, the more image frames are inserted between adjacent photo frames, the more the motion process of the subject in the adjacent photo frames can be reflected. When the number of image frames inserted between two adjacent photo frames is increased, the transition of the subject in the video between the two adjacent photo frames will be smoother. The electronic device 100 can insert different numbers of image frames in different adjacent frames to achieve different visual effects in the video.
[0115] In some embodiments, the electronic device 100 may determine the degree of excellence of each wonderful frame. The electronic device 100 may insert more image frames between the photo frame corresponding to the wonderful frame with a relatively high degree of excellence and its adjacent photo frame. The method for the electronic device 100 to determine the degree of excellence of the wonderful frame can be referred to the introduction of the subsequent embodiments, which will not be expanded here.
[0116] In some embodiments, the electronic device 100 may receive an operation by a user to change the playback duration between two adjacent photo frames. Furthermore, the electronic device 100 may determine the number of image frames to be inserted between two photo frames according to the playback duration set by the user. For example, assuming that the video generated by the electronic device 100 is played for 3 milliseconds per frame, the electronic device 100 receives an operation by a user specifying that a total of 1 second is performed from the A-th photo frame to the A+1-th photo frame. Furthermore, the electronic device 100 may determine that approximately 333 image frames need to be inserted between the A-th photo frame and the A+1-th photo frame.
[0117] Figure 4 Two specific methods for an electronic device to identify a wonderful frame are shown, and the two wonderful frame identification methods are respectively described below.
[0118] Method 1: The method for an electronic device to identify a wonderful frame may include but is not limited to the following steps:
[0119] S401: The electronic device obtains a preview frame from a preview stream.
[0120] S402: The electronic device identifies key points of the photographed object in the preview frame.
[0121] In some embodiments, the electronic device 100 can obtain one or more preview frames from the preview stream, and then identify key points of the photographed object in the preview frame. There can be multiple objects in the preview frame, and the category of each object can also be different. The category of the object can be one or more of a person, an animal, a plant, and an inanimate object.
[0122] Optionally, the key points of the photographed object in the preview frame may include skeleton points of a person or an animal. The electronic device 100 may determine the skeleton points of the photographed object in the preview frame by a skeleton point recognition algorithm. The skeleton points may include, but are not limited to, the head, neck, shoulders, elbows, wrists, waist, knees, ankles and other parts of the photographed object.
[0123] Optionally, the key points of the photographed object in the preview frame may also include key points of the face of a person or an animal. The above key points of the face may include but are not limited to: eyebrows, eyes, nose, mouth, etc. of the photographed object.
[0124] Optionally, the key points of the photographed object in the preview frame may also include key points of inanimate objects. Exemplarily, the key points may include but are not limited to: one or more points on the outline of the photographed object.
[0125] In some embodiments, the electronic device 100 can input the preview frame into the neural network model C, and then receive one or more key points output by the neural network model C.
[0126] In some embodiments, the electronic device 100 may receive a user's selection operation on a shooting object, and the electronic device 100 may identify a specific object in the preview frame in response to the user operation. For example, the electronic device 100 may respond to a user's operation of selecting "people" as a shooting object, and the electronic device 100 may identify one or more people appearing in the preview frame, and then identify the key points of each person. In other embodiments, the electronic device 100 may also filter multiple objects in the preview frame through an attention mechanism, and determine the object that the user may be most concerned about from the multiple objects captured in the preview frame. In subsequent steps, the electronic device 100 may determine one or more wonderful frames based on the key points of the object, and then generate a video.
[0127] S403: The electronic device matches the key points of the object in the preview frame with the key points of the target in the posture template.
[0128] S404: The electronic device determines that the preview frame is a wonderful frame.
[0129] One or more gesture templates may be stored in the electronic device 100 , and the gesture template may include one or more key points of the object.
[0130] In some embodiments, the electronic device 100 may calculate the similarity between the key points of the object in the posture template and the key points of the object in the preview frame. Optionally, the electronic device 100 may store a first threshold value, and when the similarity between the key points of the object in the template posture calculated by the electronic device and the key points of the object in the preview frame is greater than the first threshold value, the electronic device 100 may determine that the preview frame is a wonderful frame. Optionally, the electronic device 100 may store a second threshold value, and when the similarity between the key points of the object in the template posture and the key points of the object in the preview frame is greater than the second threshold value, it indicates that the object's motion is more complete and the frame is more wonderful.
[0131] In other embodiments, the electronic device 100 can match the key points of the object in the template posture with the key points of the object in the preview frame by a machine learning method. The electronic device 100 can input the features of the preview frame into a binary classifier. The features of the preview frame may include parameters of one or more key points of the preview frame. The binary classifier here (also referred to as a binary classification network) can be a fully connected network, and the activation function of the fully connected network may include a sigmoid function and other activation functions (such as a tanh function, a relu function, and a softmax function), and the loss function of the fully connected network may include a cross-entropy loss function and other loss functions (such as a hinge loss function, a logistic loss function, and an exponential loss function). Among them, the binary classifier can be obtained by training the features of multiple wonderful frames. The electronic device 100 can determine that the preview frame is not a wonderful frame when the binary classifier outputs 0; or when the binary classifier outputs 1, it can determine that the preview frame is a wonderful frame.
[0132] In some embodiments, the electronic device 100 may divide one or more posture templates into different template sets, each template set corresponding to a different shooting scene. The electronic device 100 may receive a user's selection operation for the above-mentioned shooting scene, and then use one or more posture templates in the template set corresponding to the shooting scene selected by the user to match the preview frame. Optionally, the above-mentioned shooting scene may also be determined by the electronic device 100 through an image processing algorithm. For example, the electronic device 100 may input the preview frame into a neural network model, and then identify the shooting scene of the preview frame image.
[0133] In some embodiments, the pose template may not be stored in the electronic device 100 but in a cloud server. The electronic device 100 may obtain the pose template from the cloud server and then match the key points of the object in the preview frame with the key points in the template pose.
[0134] In some embodiments, when the electronic device 100 determines that two preview frames in the preview stream are wonderful frames, one or more frames between the two preview frames can also be determined as wonderful frames. In other embodiments, after determining that the preview frame is a wonderful frame, the electronic device 100 can use the first Y1 frame of the preview frame and / or the last Y2 frame of the preview frame as a wonderful frame. The above Y1 and Y2 are any positive integers, and Y1 and Y2 may not be equal. In this way, the electronic device 100 can improve the fluency of the subsequently generated video.
[0135] Method 2: The method for the electronic device to identify the wonderful frame may include but is not limited to the following steps:
[0136] S411. The electronic device obtains a preview frame from the preview stream.
[0137] S412: The electronic device obtains a text description of the preview frame image content.
[0138] In some embodiments, the electronic device 100 may obtain a preview frame from the preview stream, and then obtain a text description of the preview frame image content. For example, the text description may be "a cat sitting on a chair".
[0139] Optionally, the electronic device 100 may store a neural network model D for extracting text descriptions of image content from preview frames. The electronic device 100 may input the preview frame into the neural network model D, and the neural network model D may output text descriptions of the image content in the preview frame. The neural network model D may include, but is not limited to, convolutional neural networks (CNN) models, recurrent neural network (RNN) models, long short-term memory (LSTM) models, deep neural network (DNN) models, and generative pre-training transformer (GPT) large models.
[0140] In a possible implementation, the neural network model D stored in the electronic device 100 may be a small model obtained by distilling and training a large model. The large model refers to a neural network model with a large scale, a large number of parameters, a large data set, and a complex architecture. The small model refers to a neural network model with a small scale, a small number of parameters, a small data set, and a relatively simple architecture. The electronic device 100 can process the preview frame through the small model, so that a text description of the image content in the preview frame can be obtained in a shorter time.
[0141] In another possible implementation, the neural network model D may be stored in a cloud server, and the electronic device 100 may send the preview frame to the cloud server, and then receive a text description obtained after the cloud server processes the preview frame.
[0142] S413: The electronic device determines that the text description of the preview frame contains one or more tags corresponding to the wonderful frame.
[0143] S414: The electronic device determines that the preview frame is a wonderful frame.
[0144] In some embodiments, the electronic device 100 may store one or more tags corresponding to the wonderful frames. Then, the electronic device 100 may determine that the preview frame A is a wonderful frame when the one or more tags are present in the text description of the preview frame A. Exemplarily, the text description of the preview frame A may be "a boy throws the basketball into the basket". The tags corresponding to the wonderful frames stored in the electronic device 100 may include "throw into the basket". The electronic device 100 may determine that "throw into the basket" in the text description of the preview frame A is the tag corresponding to the wonderful frame, and then determine that the preview frame A is a wonderful frame.
[0145] Optionally, the electronic device 100 may further divide the above one or more tags into different sets, each tag set corresponding to a different degree of excitement. For example, a person's slam dunk may include movements such as "jumping up", "jumping into the air", "throwing into the basket", and "falling down", and the corresponding electronic device 100 may store the tags "jumping up", "jumping into the air", "throwing into the basket", and "falling down". Among them, the "throwing into the basket" tag may belong to a tag set with a higher degree of excitement, while "jumping up", "jumping into the air", and "falling down" may belong to a tag set with a lower degree of excitement. In this way, after the electronic device 100 determines that "throwing into the basket" in the text description of preview frame A is the tag corresponding to the exciting frame, it determines preview frame A as an exciting frame with a higher degree of excitement.
[0146] In some embodiments, the electronic device 100 determines that the preview frame A is a label of a wonderful frame by the user. For example, the electronic device 100 may receive an operation of the user selecting one or more labels, and then determine that the preview frame A is a wonderful frame when the text description of the preview frame A contains one or more labels selected by the user.
[0147] In some embodiments, the electronic device 100 may store a neural network model E for determining the wonderfulness of a preview frame. The electronic device 100 may input a text description of the image content of the preview frame and the preview frame into the neural network model E, and the neural network model E may score the wonderfulness of the preview frame. The electronic device 100 may store a third threshold and / or a fourth threshold. When the wonderfulness score of the preview frame output by the neural network model E is greater than the third threshold, the electronic device 100 may determine that the preview frame is a wonderful frame. When the wonderfulness score of the preview frame output by the neural network model E is greater than the fourth threshold, the electronic device 100 may determine that the preview frame has a high degree of wonderfulness. Optionally, the neural network model may be a multimodal model.
[0148] In other embodiments, the electronic device 100 may divide one or more tags into different tag sets, each tag set corresponding to a different shooting scene. The electronic device 100 may receive a user's selection operation on a shooting scene, and then use the tag set corresponding to the shooting scene to match the text description of the preview frame. Exemplarily, the above-mentioned shooting scenes may include: sports scenes, birthday scenes, fireworks scenes, etc. Among them, the tags corresponding to the sports scenes may include but are not limited to: jumping, shooting, taking off, etc. In a possible implementation, the above-mentioned shooting scenes may also be identified by the electronic device 100 using an image processing algorithm.
[0149] In some embodiments, after the electronic device 100 determines that the preview frame is a wonderful frame, the first Y1 frame of the preview frame and / or the last Y2 frame of the preview frame can also be used as a wonderful frame. The above Y1 and Y2 are any positive integers, and Y1 and Y2 may not be equal. In this way, when the electronic device 100 determines the wonderful frame according to the text description of the preview frame, it can avoid the situation that the multiple wonderful frames determined by the electronic device 100 are discontinuous and cannot fully reflect the complete movement process of the photographed object. Taking the electronic device 100 shooting a dunking scene of a person as an example, a dunking of a person may include movements such as "jumping up", "jumping into the air", "throwing into the basket", and "falling down", but the label corresponding to the wonderful frame stored in the electronic device 100 may only include the "throwing into the basket" label. If the wonderful frame is determined only according to the "throwing into the basket" label, the wonderful frame determined by the electronic device 100 may omit the picture of the person jumping up, jumping into the air, and falling. In this way, the wonderful frames obtained by the electronic device 100 are not continuous enough, which will cause the subsequent generated video to have a relatively low fluency. Therefore, after the electronic device 100 determines that the text description of the preview frame includes "throwing into the basket", the first Y1 frame and / or the last Y2 frame of the preview frame can be used as the highlight frame. In this way, the electronic device 100 can ensure that the multiple highlight frames finally determined can include the scenes of the person jumping up, jumping into the air, throwing into the basket, and falling.
[0150] In other embodiments, the electronic device 100 may also determine the preview frames between two wonderful frames as wonderful frames. Taking the scene of a person slam dunking filmed by the electronic device 100 as an example, the labels corresponding to the wonderful frames stored in the electronic device 100 may include "jumping up", "throwing into the basket", and "falling down". If the wonderful frames are determined only based on the above three labels, the wonderful frames determined by the electronic device 100 may miss the scene when the person jumps into the air after jumping up. The electronic device can determine that the text description of preview frame A contains the label "throwing into the basket", and after determining that the text description of preview frame B contains the label "jumping up", one or more preview frames from preview frame B to preview frame can be determined as wonderful frames. In this way, the electronic device 100 can ensure that the scene when the person jumps into the air is not missed in the multiple wonderful frames finally determined.
[0151] The following describes a scenario in which an electronic device generates a video provided by an embodiment of the present application.
[0152] Figure 5A-5J The following is a schematic diagram showing a series of user interactions (UI) involved in the shooting method provided in the embodiment of the present application.
[0153] Figure 5A The home screen interface 500 on the electronic device 100 is shown as an example. Figure 5AAs shown, the main screen interface may include desktop icons of one or more application programs, which may include a desktop icon 501 of a camera application.
[0154] The electronic device 100 may detect a user operation, such as a click operation, on the icon 501. In response to the operation, the electronic device 100 may start a camera application and display a Figure 5B The photo preview interface 510 is shown. The photo preview interface 510 may include a preview window 511, a menu bar 512, a wonderful capture button 513, a shutter button 514, and a review control 515. Among them, the preview window 511 can be used to display a preview image, such as a preview image of a person playing golf. The menu bar 512 may include one or more shooting mode options, such as "night scene", "photo", and "video". The wonderful capture button 513 can be used to start the wonderful capture function. Once the user starts the wonderful capture function, the electronic device 100 will execute the shooting method provided in the embodiment of the present application, identify the wonderful frames from the preview stream, insert the photo frames corresponding to the wonderful frames, and then generate a video. The shutter button 514 is used to take photos. The review control 515 is used to open the gallery application to view photos or videos stored in the electronic device. Among them, the review control 515 can display the photos or videos recently saved by the electronic device.
[0155] The electronic device 100 may detect a user operation, such as a click operation, on the wonderful capture button 513. In response to the operation, the electronic device 100 may display a Figure 5C Dialog box 516 is shown. Dialog box 516 may include one or more shooting scene options, such as "sports" option 517 and "smiley face" option. The shooting scene options correspond to the wonderful frame types. The electronic device 100 can determine the recognition method of the wonderful frame according to the shooting scene option selected by the user.
[0156] like Figure 5C As shown in FIG. 5 , the user selects the shooting scene "sports" option 517. In response to this, the electronic device 100 can specifically start to identify the wonderful frames from the preview stream based on the wonderful frame recognition algorithm related to sports. For example, the electronic device 100 can select a posture template corresponding to sports from one or more posture templates to match the preview frame ( Figure 4 Method 1 shown). Alternatively, the electronic device can select a sports-related tag from one or more tags to match the text description of the preview frame. Similarly, when the user selects "smiley face", the electronic device 100 can select a posture template corresponding to a smiley face, or a tag related to a smiley face to match the preview frame. The shooting scenes shown in the figure are only examples and are not intended to limit the embodiments of the present application.
[0157] The electronic device 100 may detect the user's selection of the "sports" option 517. In response to the selection, the electronic device 100 may close the dialog box 516 and display the following information: Figure 5D The prompt box 518 is shown. The prompt box 518 is used to prompt the user that the wonderful capture function has been turned on.
[0158] like Figure 5E As shown in (a), (b), (c), and (d), after the electronic device recognizes the wonderful frame in the preview stream, it will generate photos corresponding to the wonderful frame and save the photos in the gallery. The review control 515 will display the photos captured by the wonderful capture function in sequence. Figure 5E As described in (d), the electronic device 100 may detect a user operation, such as a click operation, on the review control 515. In response to the operation, the electronic device 100 may open the gallery interface 520. Fig. 5F As shown, the gallery interface 520 can display one or more photos, check boxes corresponding to the photos, and a generate video button 526.
[0159] The lower left corner of the photo may display a "Great Snap" icon, indicating that the photo was captured by the Great Snap function. For example, photo 521 may be Figure 5E In the embodiment shown in (a), the electronic device 100 captures the photo; the photo 522 may be Figure 5E In the embodiment shown in (b), the electronic device 100 captures the photo; the photo 523 may be Figure 5E In the embodiment shown in (c), the electronic device 100 captures the photo; the photo 524 may be Figure 5E The electronic device 100 in the embodiment shown in (d) is captured.
[0160] The generate video button 526 is used for the electronic device 100 to generate a video using the selected photo.
[0161] like Figure 5G As shown, the electronic device 100 can detect the operation of the user selecting photo 521, photo 522, photo 523 and photo 524. In response to this operation, a check mark will be displayed in the check box corresponding to the photo, indicating that the photo is selected. The electronic device 100 can detect the operation of the user clicking the generate video button 526. In response to this operation, the electronic device 100 can generate a video using the selected photos 521, photo 522, photo 523 and photo 524. Optionally, in addition to using the photos taken by the wonderful capture function to generate a video, the electronic device 100 can also use the photos obtained by the user clicking the shutter button to generate a video.
[0162] In some embodiments, the electronic device 100 may also record the order in which the user selects the photos, and determine the time sequence of the corresponding frames of the photos in the video.
[0163] like Figure 5H As shown, in response to the user clicking the generate video button 526, the electronic device 100 may display a video editing interface 530. The video editing interface 530 may include: a video preview window 531, a toolbar 532, a frame setting area 533, a duration editing box 534, and a completion button 535. The video preview window 531 is used to preview the video generated by the electronic device 100 according to the photographed frame.
[0164] The toolbar 532 may include one or more function controls, which are used to change the playback status of the video in the video preview window 531. The function controls in the toolbar 532 may be used to fast forward, rewind, pause, start playing the video in the video preview window 531, or adjust the playback progress of the video.
[0165] The frame setting area 533 can display two adjacent photos. Figure 5H The frame setting area 533 may display the photo 521 and the photo 522 , and an arrow pointing from the photo 521 to the photo 522 .
[0166] The duration edit box 534 can receive the user's operation of changing the duration of adjacent frames, so that the electronic device 100 can change the number of image frames inserted between adjacent frames according to the duration in the duration edit box 534. The method for changing the duration of adjacent frames by the electronic device 100 can refer to the introduction of the aforementioned embodiment, which will not be repeated here. Figure 5H As shown, the duration edit box 534 can be used to set the duration of the video from the photo frame corresponding to the photo 521 to the photo frame corresponding to the photo 522, and the above duration can be 2 seconds. Among them, the duration of adjacent frames in the initial state in the duration edit box 534 can be determined by the electronic device 100 according to the wonderfulness of the photo frame corresponding to the photo 521 and the photo frame corresponding to the photo 522. The method for the electronic device 100 to determine the playback duration between adjacent wonderful frames according to the wonderfulness of the wonderful frames can refer to the introduction of the above-mentioned embodiment, which will not be repeated here.
[0167] The completion button 535 is used to store the video generated according to the corresponding photographing frame of the photo in the electronic device 100.
[0168] The electronic device 100 may detect the operation of the user sliding the frame setting area 533 to the left, and in response to the operation, the electronic device 100 may change the adjacent frames that the duration editing box 534 can control. Fig.5I As shown, the frame setting area 533 can display photo 522 and photo 523, and an arrow pointing from photo 522 to photo 523. Accordingly, the duration edit box 534 can be used to set the duration of the video from the photo frame corresponding to photo 522 to the photo frame corresponding to photo 523, and the duration can be 1 second.
[0169] In response to the user touching and clicking the completion button 535, the electronic device 100 may store the video generated based on the photographed frame. Figure 5J As shown, the electronic device 100 may display a user interface 540. The user interface 540 may be used to play a video stored in the electronic device 100 and generated according to a photographed frame corresponding to a photograph.
[0170] The electronic device needs to extract corresponding photo frames from photos 521, 522, 523, and 524, and then interpolate the photo frames to generate a video. The method for the electronic device 100 to interpolate photo frames and generate a video can refer to the introduction of the above embodiment, which will not be repeated here.
[0171] It is understandable that the electronic device 100 can identify one or more wonderful frames from the preview stream. Since the resolution of the preview frame is low, the speed at which the electronic device 100 determines the wonderful frame is also fast. In this way, the electronic device 100 can determine the wonderful frame from the preview stream in real time when the user points the camera at the target scene. The electronic device 100 can generate a photo frame corresponding to the wonderful frame and store the photo frame, so that the user can obtain one or more wonderful photos without pressing the shutter himself. The electronic device 100 can receive the operation of the user selecting a wonderful photo, so that the electronic device 100 can generate an image frame according to the photo frame corresponding to the wonderful photo selected by the user. In addition, the electronic device 100 can also change the number of image frames inserted between the photo frames corresponding to two wonderful photos in response to the user operation. In this way, the user can customize the wonderful frames that appear in the video and the transition duration between the photo frames corresponding to the two wonderful frames, thereby improving the user's experience. Finally, the electronic device 100 can generate a video according to the photo frame corresponding to the wonderful frame. Since the image quality of the photo frame is high, the image quality of each frame in the video obtained by the user is high. The user may find that the video of the wonderful clip generated by the electronic device 100 is more vivid and clear than the video of the wonderful clip directly shot.
[0172] In some embodiments, Figure 5C The illustrated embodiment is optional. The electronic device 100 can identify the shooting scene, which can reduce user operations and improve the user experience.
[0173] In some embodiments, Figure 5F-5G The embodiment shown is optional. Figure 5E In the illustrated embodiment, the user of the electronic device 100 does not need to manually select photos for generating a video.
[0174] In some embodiments, Figure 5H-Figure 5I The embodiment shown is optional. Figure 5GIn the illustrated embodiment, after the electronic device 100 receives the user's touch and click operation of generating video button 526, the video can be directly saved without the user having to adjust the playback duration between each two corresponding frames of the photos.
[0175] In some embodiments, Figure 5F-Figure 5I The embodiment shown is optional. After the electronic device 100 identifies the wonderful frame through the preview stream, it can directly generate a video based on the wonderful frame. In this way, the user only needs to point the camera at the target scene to directly obtain the wonderful frame video generated by the electronic device 100, reducing user operations. The electronic device can directly use the photo frame to generate a video without generating photos corresponding to the photo frame.
[0176] The various user interfaces described in the embodiments of the present application are only example interfaces and do not limit the present application. In other embodiments, the user interface may adopt a different interface layout, may include more or fewer controls, and may increase or decrease other functional options, as long as they are based on the same inventive concept provided by the present application, they are all within the scope of protection of the present application.
[0177] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
[0178] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.
[0179] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.
[0180] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.
Claims
1. A shooting method, the method being applied to an electronic device, the electronic device comprising a camera, It is characterized in that The method comprises: The electronic device displays a preview picture on a display screen using a preview stream, wherein the preview stream includes a plurality of preview frames; The electronic device identifies a plurality of wonderful frames from the plurality of preview frames, wherein the moving objects in the wonderful frames are in a preset motion; The electronic device obtains original images corresponding to each of the plurality of wonderful frames; The electronic device processes the original images corresponding to the multiple wonderful frames to generate multiple photo frames, and the resolution of the photo frames is greater than the resolution of the preview frames; The electronic device generates one or more insertion frames based on the multiple photo frames, wherein the multiple photo frames include adjacent first and second photo frames, adjacent third and fourth photo frames, K1 insertion frames are inserted between the first and second photo frames, and K2 insertion frames are inserted between the third and fourth photo frames, K1 and K2 are positive integers, K1 is less than K2, the third and fourth photo frames are more exciting than the first and second photo frames, and in the insertion frame between two consecutive photo frames, the position of the moving object falls on a first motion trajectory, which is a motion trajectory formed by pixel movement of the moving object in the two consecutive photo frames; The electronic device generates a video based on the multiple photographing frames and one or more inserted frames between the multiple photographing frames; The electronic device saves the video to a gallery.
2. The method according to claim 1, It is characterized in that The electronic device identifies a plurality of wonderful frames from the plurality of preview frames, including: The electronic device identifies key points of the moving object in the plurality of preview frames, where the key points of the moving object are used to represent the posture of the moving object; The electronic device identifies a plurality of wonderful frames according to key points of the moving object, and the posture of the moving object in the wonderful frames matches the first template.
3. The method according to claim 2, It is characterized in that The first template is selected by a user.
4. The method according to claim 3, It is characterized in that The first template is selected by the electronic device from a plurality of templates based on a first shooting scene, each of the plurality of templates corresponds to a different shooting scene, and the first shooting scene is the shooting scene presented in the preview screen.
5. The method according to any one of claims 1 to 4, It is characterized in that The electronic device identifies a plurality of wonderful frames from the plurality of preview frames, including: The electronic device determines a text description of each of the plurality of preview frames, where the text description is used to describe the content expressed by the preview frame; The electronic device selects a plurality of wonderful frames from a plurality of preview frames according to the text description, wherein the text descriptions of the plurality of wonderful frames contain a first keyword, and the first keyword is used to describe the preset motion.
6. The method according to any one of claims 1 to 5, It is characterized in that The electronic device also uses the preview frame between the two wonderful frames as a wonderful frame.
7. The method according to any one of claims 1 to 6, It is characterized in that The electronic device also uses the first Y1 preview frames and / or the last Y2 preview frames adjacent to the wonderful frame as wonderful frames, wherein Y1 and Y2 are positive integers.
8. The method according to any one of claims 1 to 7, It is characterized in that The electronic device displays a preview image on a display screen using a preview stream and also displays a wonderful capture button. Before the electronic device identifies multiple wonderful frames from the multiple preview frames, the electronic device detects an operation on the wonderful capture button.
9. An electronic device, It is characterized in that The electronic device includes: a display screen, a camera, a memory, and a processor coupled to the memory; the display screen is used to display an interface, the camera is used to capture images, the memory stores a computer program, and when the processor executes the computer program, the electronic device implements the method as described in any one of claims 1 to 8.
10. A computer-readable storage medium comprising computer instructions, It is characterized in that When the computer instructions are executed on an electronic device, the electronic device is caused to execute the method as claimed in any one of claims 1 to 8.
Citation Information
Patent Citations
Hot spot change process display method and device, medium and electronic device
CN112468743A
Slow action video shooting method and electronic equipment
CN112532865A
Video processing method and electronic equipment
CN115002340A
Shooting method and electronic equipment
CN115525188A
Image processing method, image processor and electronic equipment
CN115550536A
